org.apache.flink.fs.s3.common

Class FlinkS3FileSystem

java.lang.Object

org.apache.flink.core.fs.FileSystem

org.apache.flink.runtime.fs.hdfs.HadoopFileSystem

org.apache.flink.fs.s3.common.FlinkS3FileSystem

All Implemented Interfaces:

EntropyInjectingFileSystem

public class FlinkS3FileSystem
extends HadoopFileSystem
implements EntropyInjectingFileSystem

Implementation of the Flink FileSystem interface for S3. This class implements the common behavior implemented directly by Flink and delegates common calls to an implementation of Hadoop's filesystem abstraction.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.core.fs.FileSystem

FileSystem.WriteMode

Field Summary

Fields

Modifier and Type	Field and Description
static long	S3_MULTIPART_MIN_PART_SIZE The minimum size of a part in the multipart upload, except for the last part: 5 MIBytes.

Constructor Summary

Constructors

Constructor and Description
FlinkS3FileSystem(org.apache.hadoop.fs.FileSystem hadoopS3FileSystem, String localTmpDirectory, String entropyInjectionKey, int entropyLength, S3AccessHelper s3UploadHelper, long s3uploadPartSize, int maxConcurrentUploadsPerStream) Creates a FlinkS3FileSystem based on the given Hadoop S3 file system.

Method Summary

Modifier and Type	Method and Description
RecoverableWriter	createRecoverableWriter() Creates a new RecoverableWriter.
String	generateEntropy() Creates a string with random entropy to be injected into a path.
String	getEntropyInjectionKey() Gets the marker string that represents the substring of a path to be replaced by the entropy characters.
FileSystemKind	getKind() Gets a description of the characteristics of this file system.
String	getLocalTmpDir()

Methods inherited from class org.apache.flink.runtime.fs.hdfs.HadoopFileSystem

create, create, delete, exists, getDefaultBlockSize, getFileBlockLocations, getFileStatus, getHadoopFileSystem, getHomeDirectory, getUri, getWorkingDirectory, isDistributedFS, listStatus, mkdirs, open, open, rename, toHadoopPath

Methods inherited from class org.apache.flink.core.fs.FileSystem

create, get, getDefaultFsUri, getLocalFileSystem, getUnguardedFileSystem, initialize, initOutPathDistFS, initOutPathLocalFS

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

S3_MULTIPART_MIN_PART_SIZE

public static final long S3_MULTIPART_MIN_PART_SIZE

The minimum size of a part in the multipart upload, except for the last part: 5 MIBytes.

See Also:

Constant Field Values

Constructor Detail

FlinkS3FileSystem

public FlinkS3FileSystem(org.apache.hadoop.fs.FileSystem hadoopS3FileSystem,
                         String localTmpDirectory,
                         @Nullable
                         String entropyInjectionKey,
                         int entropyLength,
                         @Nullable
                         S3AccessHelper s3UploadHelper,
                         long s3uploadPartSize,
                         int maxConcurrentUploadsPerStream)

Creates a FlinkS3FileSystem based on the given Hadoop S3 file system. The given Hadoop file system object is expected to be initialized already.

This constructor additionally configures the entropy injection for the file system.

Parameters:

hadoopS3FileSystem - The Hadoop FileSystem that will be used under the hood.

entropyInjectionKey - The substring that will be replaced by entropy or removed.

entropyLength - The number of random alphanumeric characters to inject as entropy.

Method Detail

getEntropyInjectionKey

@Nullable
public String getEntropyInjectionKey()

Description copied from interface: EntropyInjectingFileSystem

Gets the marker string that represents the substring of a path to be replaced by the entropy characters.

Specified by:

getEntropyInjectionKey in interface EntropyInjectingFileSystem

generateEntropy

public String generateEntropy()

Description copied from interface: EntropyInjectingFileSystem

Creates a string with random entropy to be injected into a path.

Specified by:

generateEntropy in interface EntropyInjectingFileSystem

getKind

public FileSystemKind getKind()

Description copied from class: FileSystem

Gets a description of the characteristics of this file system.

Overrides:

getKind in class HadoopFileSystem

getLocalTmpDir

public String getLocalTmpDir()

createRecoverableWriter

public RecoverableWriter createRecoverableWriter()
                                          throws IOException

Description copied from class: FileSystem

Creates a new RecoverableWriter. A recoverable writer creates streams that can persist and recover their intermediate state. Persisting and recovering intermediate state is a core building block for writing to files that span multiple checkpoints.

The returned object can act as a shared factory to open and recover multiple streams.

This method is optional on file systems and various file system implementations may not support this method, throwing an UnsupportedOperationException.

Overrides:

createRecoverableWriter in class HadoopFileSystem

Returns:

A RecoverableWriter for this file system.

Throws:

IOException - Thrown, if the recoverable writer cannot be instantiated.