org.apache.flink.fs.s3.common

Class AbstractS3FileSystemFactory

java.lang.Object

org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory

All Implemented Interfaces:

FileSystemFactory, Plugin

Direct Known Subclasses:

S3FileSystemFactory, S3FileSystemFactory

public abstract class AbstractS3FileSystemFactory
extends Object
implements FileSystemFactory

Base class for file system factories that create S3 file systems.

Field Summary

Fields

Modifier and Type	Field and Description
static ConfigOption<String>	ENTROPY_INJECT_KEY_OPTION The substring to be replaced by random entropy in checkpoint paths.
static ConfigOption<Integer>	ENTROPY_INJECT_LENGTH_OPTION The number of entropy characters, in case entropy injection is configured.
static ConfigOption<Integer>	MAX_CONCURRENT_UPLOADS
static ConfigOption<Long>	PART_UPLOAD_MIN_SIZE

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	AbstractS3FileSystemFactory(String name, HadoopConfigLoader hadoopConfigLoader)

Method Summary

Modifier and Type	Method and Description
void	configure(Configuration config) Optional method for plugins to pick up settings from the configuration.
FileSystem	create(URI fsUri) Creates a new file system for the given file system URI.
protected FileSystem	createFlinkFileSystem(org.apache.hadoop.fs.FileSystem fs, String localTmpDirectory, String entropyInjectionKey, int numEntropyChars, S3AccessHelper s3AccessHelper, long s3minPartSize, int maxConcurrentUploads)
protected abstract org.apache.hadoop.fs.FileSystem	createHadoopFileSystem()
protected abstract URI	getInitURI(URI fsUri, org.apache.hadoop.conf.Configuration hadoopConfig)
protected abstract S3AccessHelper	getS3AccessHelper(org.apache.hadoop.fs.FileSystem fs)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.core.fs.FileSystemFactory

getScheme

Methods inherited from interface org.apache.flink.core.plugin.Plugin

getClassLoader

Field Detail

PART_UPLOAD_MIN_SIZE

public static final ConfigOption<Long> PART_UPLOAD_MIN_SIZE

MAX_CONCURRENT_UPLOADS

public static final ConfigOption<Integer> MAX_CONCURRENT_UPLOADS

ENTROPY_INJECT_KEY_OPTION

public static final ConfigOption<String> ENTROPY_INJECT_KEY_OPTION

The substring to be replaced by random entropy in checkpoint paths.

ENTROPY_INJECT_LENGTH_OPTION

public static final ConfigOption<Integer> ENTROPY_INJECT_LENGTH_OPTION

The number of entropy characters, in case entropy injection is configured.

Constructor Detail

AbstractS3FileSystemFactory

protected AbstractS3FileSystemFactory(String name,
                                      HadoopConfigLoader hadoopConfigLoader)

Method Detail

configure

public void configure(Configuration config)

Description copied from interface: Plugin

Optional method for plugins to pick up settings from the configuration.

Specified by:

configure in interface Plugin

Parameters:

config - The configuration to apply to the plugin.

create

public FileSystem create(URI fsUri)
                  throws IOException

Description copied from interface: FileSystemFactory

Creates a new file system for the given file system URI. The URI describes the type of file system (via its scheme) and optionally the authority (for example the host) of the file system.

Specified by:

create in interface FileSystemFactory

Parameters:

fsUri - The URI that describes the file system.

Returns:

A new instance of the specified file system.

Throws:

IOException - Thrown if the file system could not be instantiated.

createFlinkFileSystem

protected FileSystem createFlinkFileSystem(org.apache.hadoop.fs.FileSystem fs,
                                           String localTmpDirectory,
                                           String entropyInjectionKey,
                                           int numEntropyChars,
                                           S3AccessHelper s3AccessHelper,
                                           long s3minPartSize,
                                           int maxConcurrentUploads)

createHadoopFileSystem

protected abstract org.apache.hadoop.fs.FileSystem createHadoopFileSystem()

getInitURI

protected abstract URI getInitURI(URI fsUri,
                                  org.apache.hadoop.conf.Configuration hadoopConfig)

getS3AccessHelper

@Nullable
protected abstract S3AccessHelper getS3AccessHelper(org.apache.hadoop.fs.FileSystem fs)