Package org.apache.flink.fs.s3.common

Class AbstractS3FileSystemFactory

java.lang.Object

org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory

All Implemented Interfaces:

FileSystemFactory, Plugin

Direct Known Subclasses:

S3FileSystemFactory, S3FileSystemFactory

public abstract class AbstractS3FileSystemFactory
extends Object
implements FileSystemFactory

Base class for file system factories that create S3 file systems.

Field Summary

Fields

Modifier and Type	Field	Description
static ConfigOption<String>	ACCESS_KEY
static ConfigOption<String>	ENDPOINT
static ConfigOption<String>	ENTROPY_INJECT_KEY_OPTION	The substring to be replaced by random entropy in checkpoint paths.
static ConfigOption<Integer>	ENTROPY_INJECT_LENGTH_OPTION	The number of entropy characters, in case entropy injection is configured.
static ConfigOption<Integer>	MAX_CONCURRENT_UPLOADS
static ConfigOption<Long>	PART_UPLOAD_MIN_SIZE
static ConfigOption<Integer>	S5CMD_BATCH_MAX_FILES
static ConfigOption<MemorySize>	S5CMD_BATCH_MAX_SIZE
static ConfigOption<String>	S5CMD_EXTRA_ARGS
static ConfigOption<String>	S5CMD_PATH
static ConfigOption<String>	SECRET_KEY

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	AbstractS3FileSystemFactory(String name, HadoopConfigLoader hadoopConfigLoader)

Method Summary

Modifier and Type	Method	Description
void	configure(Configuration config)	Optional method for plugins to pick up settings from the configuration.
FileSystem	create(URI fsUri)	Creates a new file system for the given file system URI.
protected FileSystem	createFlinkFileSystem(org.apache.hadoop.fs.FileSystem fs, FlinkS3FileSystem.S5CmdConfiguration s5CmdConfiguration, String localTmpDirectory, String entropyInjectionKey, int numEntropyChars, S3AccessHelper s3AccessHelper, long s3minPartSize, int maxConcurrentUploads)
protected abstract org.apache.hadoop.fs.FileSystem	createHadoopFileSystem()
protected abstract URI	getInitURI(URI fsUri, org.apache.hadoop.conf.Configuration hadoopConfig)
protected abstract S3AccessHelper	getS3AccessHelper(org.apache.hadoop.fs.FileSystem fs)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.core.fs.FileSystemFactory

getScheme

Methods inherited from interface org.apache.flink.core.plugin.Plugin

getClassLoader

Field Detail

ACCESS_KEY

public static final ConfigOption<String> ACCESS_KEY

SECRET_KEY

public static final ConfigOption<String> SECRET_KEY

ENDPOINT

public static final ConfigOption<String> ENDPOINT

S5CMD_PATH

public static final ConfigOption<String> S5CMD_PATH

S5CMD_EXTRA_ARGS

public static final ConfigOption<String> S5CMD_EXTRA_ARGS

S5CMD_BATCH_MAX_SIZE

public static final ConfigOption<MemorySize> S5CMD_BATCH_MAX_SIZE

S5CMD_BATCH_MAX_FILES

public static final ConfigOption<Integer> S5CMD_BATCH_MAX_FILES

PART_UPLOAD_MIN_SIZE

public static final ConfigOption<Long> PART_UPLOAD_MIN_SIZE

MAX_CONCURRENT_UPLOADS

public static final ConfigOption<Integer> MAX_CONCURRENT_UPLOADS

ENTROPY_INJECT_KEY_OPTION

public static final ConfigOption<String> ENTROPY_INJECT_KEY_OPTION

The substring to be replaced by random entropy in checkpoint paths.

ENTROPY_INJECT_LENGTH_OPTION

public static final ConfigOption<Integer> ENTROPY_INJECT_LENGTH_OPTION

The number of entropy characters, in case entropy injection is configured.

Constructor Detail

AbstractS3FileSystemFactory

protected AbstractS3FileSystemFactory(String name,
                                      HadoopConfigLoader hadoopConfigLoader)

Method Detail

configure

public void configure(Configuration config)

Description copied from interface: Plugin

Optional method for plugins to pick up settings from the configuration.

Specified by:

configure in interface Plugin

Parameters:

config - The configuration to apply to the plugin.

create

public FileSystem create(URI fsUri)
                  throws IOException

Description copied from interface: FileSystemFactory

Creates a new file system for the given file system URI. The URI describes the type of file system (via its scheme) and optionally the authority (for example the host) of the file system.

Specified by:

create in interface FileSystemFactory

Parameters:

fsUri - The URI that describes the file system.

Returns:

A new instance of the specified file system.

Throws:

IOException - Thrown if the file system could not be instantiated.

createFlinkFileSystem

protected FileSystem createFlinkFileSystem(org.apache.hadoop.fs.FileSystem fs,
                                           @Nullable
                                           FlinkS3FileSystem.S5CmdConfiguration s5CmdConfiguration,
                                           String localTmpDirectory,
                                           @Nullable
                                           String entropyInjectionKey,
                                           int numEntropyChars,
                                           @Nullable
                                           S3AccessHelper s3AccessHelper,
                                           long s3minPartSize,
                                           int maxConcurrentUploads)

createHadoopFileSystem

protected abstract org.apache.hadoop.fs.FileSystem createHadoopFileSystem()

getInitURI

protected abstract URI getInitURI(URI fsUri,
                                  org.apache.hadoop.conf.Configuration hadoopConfig)

getS3AccessHelper

@Nullable
protected abstract S3AccessHelper getS3AccessHelper(org.apache.hadoop.fs.FileSystem fs)