public class FlinkS3FileSystem extends HadoopFileSystem implements EntropyInjectingFileSystem
FileSystem
interface for S3. This
class implements the common behavior implemented directly by Flink and delegates common calls to
an implementation of Hadoop's filesystem abstraction.FileSystem.WriteMode
Modifier and Type | Field and Description |
---|---|
static long |
S3_MULTIPART_MIN_PART_SIZE
The minimum size of a part in the multipart upload, except for the last part: 5 MIBytes.
|
Constructor and Description |
---|
FlinkS3FileSystem(org.apache.hadoop.fs.FileSystem hadoopS3FileSystem,
String localTmpDirectory,
String entropyInjectionKey,
int entropyLength,
S3AccessHelper s3UploadHelper,
long s3uploadPartSize,
int maxConcurrentUploadsPerStream)
Creates a FlinkS3FileSystem based on the given Hadoop S3 file system.
|
Modifier and Type | Method and Description |
---|---|
RecoverableWriter |
createRecoverableWriter()
Creates a new
RecoverableWriter . |
String |
generateEntropy()
Creates a string with random entropy to be injected into a path.
|
String |
getEntropyInjectionKey()
Gets the marker string that represents the substring of a path to be replaced by the entropy
characters.
|
FileSystemKind |
getKind()
Gets a description of the characteristics of this file system.
|
String |
getLocalTmpDir() |
create, create, delete, exists, getDefaultBlockSize, getFileBlockLocations, getFileStatus, getHadoopFileSystem, getHomeDirectory, getUri, getWorkingDirectory, isDistributedFS, listStatus, mkdirs, open, open, rename, toHadoopPath
create, get, getDefaultFsUri, getLocalFileSystem, getUnguardedFileSystem, initialize, initialize, initOutPathDistFS, initOutPathLocalFS
public static final long S3_MULTIPART_MIN_PART_SIZE
public FlinkS3FileSystem(org.apache.hadoop.fs.FileSystem hadoopS3FileSystem, String localTmpDirectory, @Nullable String entropyInjectionKey, int entropyLength, @Nullable S3AccessHelper s3UploadHelper, long s3uploadPartSize, int maxConcurrentUploadsPerStream)
This constructor additionally configures the entropy injection for the file system.
hadoopS3FileSystem
- The Hadoop FileSystem that will be used under the hood.entropyInjectionKey
- The substring that will be replaced by entropy or removed.entropyLength
- The number of random alphanumeric characters to inject as entropy.@Nullable public String getEntropyInjectionKey()
EntropyInjectingFileSystem
You can disable entropy injection if you return null here.
getEntropyInjectionKey
in interface EntropyInjectingFileSystem
public String generateEntropy()
EntropyInjectingFileSystem
generateEntropy
in interface EntropyInjectingFileSystem
public FileSystemKind getKind()
FileSystem
getKind
in class HadoopFileSystem
public String getLocalTmpDir()
public RecoverableWriter createRecoverableWriter() throws IOException
FileSystem
RecoverableWriter
. A recoverable writer creates streams that can
persist and recover their intermediate state. Persisting and recovering intermediate state is
a core building block for writing to files that span multiple checkpoints.
The returned object can act as a shared factory to open and recover multiple streams.
This method is optional on file systems and various file system implementations may not
support this method, throwing an UnsupportedOperationException
.
createRecoverableWriter
in class HadoopFileSystem
IOException
- Thrown, if the recoverable writer cannot be instantiated.Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.