Modifier and Type | Method and Description |
---|---|
protected FileInputFormat.FileBaseStatistics |
FileInputFormat.getFileStats(FileInputFormat.FileBaseStatistics cachedStats,
Path filePath,
FileSystem fs,
ArrayList<FileStatus> files) |
Constructor and Description |
---|
DuplicatingStateChangeFsUploader(JobID jobID,
Path basePath,
FileSystem fileSystem,
boolean compression,
int bufferSize,
ChangelogStorageMetricGroup metrics,
TaskChangelogRegistry changelogRegistry,
LocalRecoveryDirectoryProvider localRecoveryDirectoryProvider) |
StateChangeFsUploader(JobID jobID,
Path basePath,
FileSystem fileSystem,
boolean compression,
int bufferSize,
ChangelogStorageMetricGroup metrics,
TaskChangelogRegistry changelogRegistry) |
StateChangeFsUploader(JobID jobID,
Path basePath,
FileSystem fileSystem,
boolean compression,
int bufferSize,
ChangelogStorageMetricGroup metrics,
TaskChangelogRegistry changelogRegistry,
java.util.function.BiFunction<Path,Long,StreamStateHandle> handleFactory) |
Modifier and Type | Method and Description |
---|---|
protected void |
NonSplittingRecursiveEnumerator.convertToSourceSplits(FileStatus file,
FileSystem fs,
List<FileSourceSplit> target) |
protected void |
BlockSplittingRecursiveEnumerator.convertToSourceSplits(FileStatus file,
FileSystem fs,
List<FileSourceSplit> target) |
Modifier and Type | Method and Description |
---|---|
FileSystem |
FileSystemFactory.create(URI fsUri)
Creates a new file system for the given file system URI.
|
Modifier and Type | Method and Description |
---|---|
static List<Path> |
PartitionTempFileManager.listTaskTemporaryPaths(FileSystem fs,
Path basePath)
Returns task temporary paths in this checkpoint.
|
Modifier and Type | Method and Description |
---|---|
List<PartitionCommitPolicy> |
PartitionCommitPolicyFactory.createPolicyChain(ClassLoader cl,
java.util.function.Supplier<FileSystem> fsSupplier)
Create a policy chain.
|
Constructor and Description |
---|
PartitionLoader(boolean overwrite,
FileSystem sourceFs,
TableMetaStoreFactory factory,
boolean isToLocal,
ObjectIdentifier identifier,
List<PartitionCommitPolicy> policies) |
SuccessFileCommitPolicy(String fileName,
FileSystem fileSystem) |
Modifier and Type | Method and Description |
---|---|
FileSystem |
CompactContext.getFileSystem() |
FileSystem |
CompactContext.CompactContextImpl.getFileSystem() |
Modifier and Type | Method and Description |
---|---|
static CompactContext |
CompactContext.create(Configuration config,
FileSystem fileSystem,
String partition,
Path path) |
Constructor and Description |
---|
CompactCoordinator(SupplierWithException<FileSystem,IOException> fsFactory,
long targetFileSize) |
CompactOperator(SupplierWithException<FileSystem,IOException> fsFactory,
CompactReader.Factory<T> readerFactory,
CompactWriter.Factory<T> writerFactory) |
Modifier and Type | Method and Description |
---|---|
FileSystem |
HadoopFileSystemFactory.create(URI uri) |
Modifier and Type | Class and Description |
---|---|
class |
LimitedConnectionsFileSystem
A file system that limits the number of concurrently open input streams, output streams, and
total streams for a target file system.
|
class |
SafetyNetWrapperFileSystem
This is a
WrappingProxy around FileSystem which (i) wraps all opened streams as
ClosingFSDataInputStream or ClosingFSDataOutputStream and (ii) registers them to
a SafetyNetCloseableRegistry . |
Modifier and Type | Method and Description |
---|---|
FileSystem |
PluginFileSystemFactory.create(URI fsUri) |
FileSystem |
FileSystemFactory.create(URI fsUri)
Creates a new file system for the given file system URI.
|
FileSystem |
ConnectionLimitingFactory.create(URI fsUri) |
static FileSystem |
FileSystem.get(URI uri)
Returns a reference to the
FileSystem instance for accessing the file system
identified by the given URI . |
FileSystem |
Path.getFileSystem()
Returns the FileSystem that owns this Path.
|
static FileSystem |
FileSystem.getLocalFileSystem()
Returns a reference to the
FileSystem instance for accessing the local file system. |
static FileSystem |
FileSystem.getUnguardedFileSystem(URI fsUri) |
FileSystem |
SafetyNetWrapperFileSystem.getWrappedDelegate() |
Modifier and Type | Method and Description |
---|---|
static Path |
EntropyInjector.addEntropy(FileSystem fs,
Path path)
Handles entropy injection across regular and entropy-aware file systems.
|
static OutputStreamAndPath |
EntropyInjector.createEntropyAware(FileSystem fs,
Path path,
FileSystem.WriteMode writeMode)
Handles entropy injection across regular and entropy-aware file systems.
|
static boolean |
EntropyInjector.isEntropyInjecting(FileSystem fs) |
Path |
Path.makeQualified(FileSystem fs)
Returns a qualified path object.
|
static Path |
EntropyInjector.removeEntropyMarkerIfPresent(FileSystem fs,
Path path)
Removes the entropy marker string from the path, if the given file system is an
entropy-injecting file system (implements
EntropyInjectingFileSystem ) and the entropy
marker key is present. |
Constructor and Description |
---|
LimitedConnectionsFileSystem(FileSystem originalFs,
int maxNumOpenStreamsTotal)
Creates a new output connection limiting file system.
|
LimitedConnectionsFileSystem(FileSystem originalFs,
int maxNumOpenStreamsTotal,
int maxNumOpenOutputStreams,
int maxNumOpenInputStreams,
long streamOpenTimeout,
long streamInactivityTimeout)
Creates a new output connection limiting file system, limiting input and output streams with
potentially different quotas.
|
LimitedConnectionsFileSystem(FileSystem originalFs,
int maxNumOpenStreamsTotal,
long streamOpenTimeout,
long streamInactivityTimeout)
Creates a new output connection limiting file system.
|
SafetyNetWrapperFileSystem(FileSystem unsafeFileSystem,
SafetyNetCloseableRegistry registry) |
Modifier and Type | Class and Description |
---|---|
class |
LocalFileSystem
The class
LocalFileSystem is an implementation of the FileSystem interface for
the local file system of the machine where the JVM runs. |
Modifier and Type | Method and Description |
---|---|
FileSystem |
LocalFileSystemFactory.create(URI fsUri) |
Constructor and Description |
---|
LocalFileStatus(File f,
FileSystem fs)
Creates a
LocalFileStatus object from a given File object. |
Modifier and Type | Method and Description |
---|---|
FileSystem |
AnotherDummyFSFactory.create(URI fsUri) |
Modifier and Type | Method and Description |
---|---|
FileSystem |
AbstractAzureFSFactory.create(URI fsUri) |
Modifier and Type | Method and Description |
---|---|
FileSystem |
DummyFSFactory.create(URI fsUri) |
Modifier and Type | Method and Description |
---|---|
FileSystem |
GSFileSystemFactory.create(URI fsUri) |
Modifier and Type | Class and Description |
---|---|
class |
FlinkOSSFileSystem
Implementation of the Flink
FileSystem interface for Aliyun OSS. |
Modifier and Type | Method and Description |
---|---|
FileSystem |
OSSFileSystemFactory.create(URI fsUri) |
Modifier and Type | Class and Description |
---|---|
class |
FlinkS3FileSystem
Implementation of the Flink
FileSystem interface for S3. |
Modifier and Type | Method and Description |
---|---|
FileSystem |
AbstractS3FileSystemFactory.create(URI fsUri) |
protected FileSystem |
AbstractS3FileSystemFactory.createFlinkFileSystem(org.apache.hadoop.fs.FileSystem fs,
String localTmpDirectory,
String entropyInjectionKey,
int numEntropyChars,
S3AccessHelper s3AccessHelper,
long s3minPartSize,
int maxConcurrentUploads) |
Modifier and Type | Class and Description |
---|---|
class |
FlinkS3PrestoFileSystem
FlinkS3PrestoFileSystem provides custom recursive deletion functionality to work around a
bug in the internally used Presto file system. |
Modifier and Type | Method and Description |
---|---|
protected FileSystem |
S3FileSystemFactory.createFlinkFileSystem(org.apache.hadoop.fs.FileSystem fs,
String localTmpDirectory,
String entropyInjectionKey,
int numEntropyChars,
S3AccessHelper s3AccessHelper,
long s3minPartSize,
int maxConcurrentUploads) |
Constructor and Description |
---|
FileSystemBlobStore(FileSystem fileSystem,
String storagePath) |
Modifier and Type | Class and Description |
---|---|
class |
HadoopFileSystem
A
FileSystem that wraps an Hadoop File System . |
Modifier and Type | Method and Description |
---|---|
FileSystem |
HadoopFsFactory.create(URI fsUri) |
Modifier and Type | Method and Description |
---|---|
protected CheckpointStorageLocation |
FsCheckpointStorageAccess.createSavepointLocation(FileSystem fs,
Path location) |
protected abstract CheckpointStorageLocation |
AbstractFsCheckpointStorageAccess.createSavepointLocation(FileSystem fs,
Path location) |
Constructor and Description |
---|
FileBasedStateOutputStream(FileSystem fileSystem,
Path path) |
FsCheckpointMetadataOutputStream(FileSystem fileSystem,
Path metadataFilePath,
Path exclusiveCheckpointDir) |
FsCheckpointStateOutputStream(Path basePath,
FileSystem fs,
int bufferSize,
int localStateThreshold) |
FsCheckpointStateOutputStream(Path basePath,
FileSystem fs,
int bufferSize,
int localStateThreshold,
boolean allowRelativePaths) |
FsCheckpointStorageAccess(FileSystem fs,
Path checkpointBaseDirectory,
Path defaultSavepointDirectory,
JobID jobId,
int fileSizeThreshold,
int writeBufferSize) |
FsCheckpointStorageLocation(FileSystem fileSystem,
Path checkpointDir,
Path sharedStateDir,
Path taskOwnedStateDir,
CheckpointStorageLocationReference reference,
int fileStateSizeThreshold,
int writeBufferSize) |
FsCheckpointStreamFactory(FileSystem fileSystem,
Path checkpointDirectory,
Path sharedStateDirectory,
int fileStateSizeThreshold,
int writeBufferSize)
Creates a new stream factory that stores its checkpoint data in the file system and location
defined by the given Path.
|
FsCompletedCheckpointStorageLocation(FileSystem fs,
Path exclusiveCheckpointDir,
FileStateHandle metadataFileHandle,
String externalPointer) |
FSDataOutputStreamWrapper(FileSystem fileSystem,
Path metadataFilePath) |
Modifier and Type | Method and Description |
---|---|
protected CheckpointStorageLocation |
MemoryBackendCheckpointStorageAccess.createSavepointLocation(FileSystem fs,
Path location) |
Constructor and Description |
---|
PersistentMetadataCheckpointStorageLocation(FileSystem fileSystem,
Path checkpointDir,
int maxStateSize)
Creates a checkpoint storage persists metadata to a file system and stores state in line in
state handles with the metadata.
|
Modifier and Type | Method and Description |
---|---|
static FileStatus[] |
PartitionPathUtils.listStatusWithoutHidden(FileSystem fs,
Path dir)
List file status without hidden files.
|
static List<Tuple2<LinkedHashMap<String,String>,Path>> |
PartitionPathUtils.searchPartSpecAndPaths(FileSystem fs,
Path path,
int partitionNumber)
Search all partitions in this path.
|
Copyright © 2014–2022 The Apache Software Foundation. All rights reserved.