org.apache.flink.configuration

Class CoreOptions

java.lang.Object

org.apache.flink.configuration.CoreOptions

@PublicEvolving
public class CoreOptions
extends Object

The set of configuration options for core parameters.

Field Summary

Fields

Modifier and Type	Field and Description
static ConfigOption<String>	ALLOWED_FALLBACK_FILESYSTEMS
static ConfigOption<List<String>>	ALWAYS_PARENT_FIRST_LOADER_PATTERNS The namespace patterns for classes that are loaded with a preference from the parent classloader, meaning the application class path, rather than any user code jar file.
static ConfigOption<List<String>>	ALWAYS_PARENT_FIRST_LOADER_PATTERNS_ADDITIONAL
static ConfigOption<Boolean>	CHECK_LEAKED_CLASSLOADER
static ConfigOption<String>	CLASSLOADER_RESOLVE_ORDER Defines the class resolution strategy when loading classes from user code, meaning whether to first check the user code jar ("child-first") or the application classpath ("parent-first")
static ConfigOption<String>	DEFAULT_FILESYSTEM_SCHEME The default filesystem scheme, used for paths that do not declare a scheme explicitly.
static ConfigOption<Integer>	DEFAULT_PARALLELISM
static ConfigOption<Boolean>	FAIL_ON_USER_CLASS_LOADING_METASPACE_OOM
static ConfigOption<Boolean>	FILESYSTEM_OUTPUT_ALWAYS_CREATE_DIRECTORY Specifies whether the file systems should always create a directory for the output, even with a parallelism of one.
static ConfigOption<Boolean>	FILESYTEM_DEFAULT_OVERRIDE Specifies whether file output writers should overwrite existing files by default.
static ConfigOption<String>	FLINK_CLI_JVM_OPTIONS
static ConfigOption<String>	FLINK_HADOOP_CONF_DIR This options is here only for documentation generation, it is only evaluated in the shell scripts.
static ConfigOption<String>	FLINK_HBASE_CONF_DIR This options is here only for documentation generation, it is only evaluated in the shell scripts.
static ConfigOption<String>	FLINK_HS_JVM_OPTIONS
static ConfigOption<String>	FLINK_JM_JVM_OPTIONS
static ConfigOption<String>	FLINK_JVM_OPTIONS
static ConfigOption<String>	FLINK_LOG_DIR This options is here only for documentation generation, it is only evaluated in the shell scripts.
static ConfigOption<Integer>	FLINK_LOG_MAX This options is here only for documentation generation, it is only evaluated in the shell scripts.
static ConfigOption<String>	FLINK_PID_DIR The config parameter defining the directory for Flink PID file.
static ConfigOption<String>	FLINK_SSH_OPTIONS This options is here only for documentation generation, it is only evaluated in the shell scripts.
static ConfigOption<String>	FLINK_TM_JVM_OPTIONS
static ConfigOption<String>	FLINK_YARN_CONF_DIR This options is here only for documentation generation, it is only evaluated in the shell scripts.
static String[]	PARENT_FIRST_LOGGING_PATTERNS
static ConfigOption<List<String>>	PLUGIN_ALWAYS_PARENT_FIRST_LOADER_PATTERNS Plugin-specific option of ALWAYS_PARENT_FIRST_LOADER_PATTERNS.
static ConfigOption<List<String>>	PLUGIN_ALWAYS_PARENT_FIRST_LOADER_PATTERNS_ADDITIONAL
static ConfigOption<String>	TMP_DIRS The config parameter defining the directories for temporary files, separated by ",", "|", or the system's File.pathSeparator.

Constructor Summary

Constructors

Constructor and Description
CoreOptions()

Method Summary

Modifier and Type	Method and Description
static ConfigOption<Integer>	fileSystemConnectionLimit(String scheme) The total number of input plus output connections that a file system for the given scheme may open.
static ConfigOption<Integer>	fileSystemConnectionLimitIn(String scheme) The total number of input connections that a file system for the given scheme may open.
static ConfigOption<Integer>	fileSystemConnectionLimitOut(String scheme) The total number of output connections that a file system for the given scheme may open.
static ConfigOption<Long>	fileSystemConnectionLimitStreamInactivityTimeout(String scheme) If any connection limit is configured, this option can be optionally set to define after which time (in milliseconds) inactive streams are reclaimed.
static ConfigOption<Long>	fileSystemConnectionLimitTimeout(String scheme) If any connection limit is configured, this option can be optionally set to define after which time (in milliseconds) stream opening fails with a timeout exception, if no stream connection becomes available.
static String[]	getParentFirstLoaderPatterns(Configuration config)
static String[]	getPluginParentFirstLoaderPatterns(Configuration config)
static String[]	mergeListsToArray(List<String> base, List<String> append)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PARENT_FIRST_LOGGING_PATTERNS

@Internal
public static final String[] PARENT_FIRST_LOGGING_PATTERNS

CLASSLOADER_RESOLVE_ORDER

public static final ConfigOption<String> CLASSLOADER_RESOLVE_ORDER

Defines the class resolution strategy when loading classes from user code, meaning whether to first check the user code jar ("child-first") or the application classpath ("parent-first")

The default settings indicate to load classes first from the user code jar, which means that user code jars can include and load different dependencies than Flink uses (transitively).

Exceptions to the rules are defined via ALWAYS_PARENT_FIRST_LOADER_PATTERNS.

ALWAYS_PARENT_FIRST_LOADER_PATTERNS

public static final ConfigOption<List<String>> ALWAYS_PARENT_FIRST_LOADER_PATTERNS

The namespace patterns for classes that are loaded with a preference from the parent classloader, meaning the application class path, rather than any user code jar file. This option only has an effect when CLASSLOADER_RESOLVE_ORDER is set to "child-first".

It is important that all classes whose objects move between Flink's runtime and any user code (including Flink connectors that run as part of the user code) are covered by these patterns here. Otherwise it is be possible that the Flink runtime and the user code load two different copies of a class through the different class loaders. That leads to errors like "X cannot be cast to X" exceptions, where both class names are equal, or "X cannot be assigned to Y", where X should be a subclass of Y.

The following classes are loaded parent-first, to avoid any duplication:

• All core Java classes (java.*), because they must never be duplicated.
• All core Scala classes (scala.*). Currently Scala is used in the Flink runtime and in the user code, and some Scala classes cross the boundary, such as the FunctionX classes. That may change if Scala eventually lives purely as part of the user code.
• All Flink classes (org.apache.flink.*). Note that this means that connectors and formats (flink-avro, etc) are loaded parent-first as well if they are in the core classpath.
• Java annotations and loggers, defined by the following list: javax.annotation;org.slf4j;org.apache.log4j;org.apache.logging;org.apache.commons.logging;ch.qos.logback. This is done for convenience, to avoid duplication of annotations and multiple log bindings.

ALWAYS_PARENT_FIRST_LOADER_PATTERNS_ADDITIONAL

public static final ConfigOption<List<String>> ALWAYS_PARENT_FIRST_LOADER_PATTERNS_ADDITIONAL

FAIL_ON_USER_CLASS_LOADING_METASPACE_OOM

public static final ConfigOption<Boolean> FAIL_ON_USER_CLASS_LOADING_METASPACE_OOM

CHECK_LEAKED_CLASSLOADER

public static final ConfigOption<Boolean> CHECK_LEAKED_CLASSLOADER

PLUGIN_ALWAYS_PARENT_FIRST_LOADER_PATTERNS

public static final ConfigOption<List<String>> PLUGIN_ALWAYS_PARENT_FIRST_LOADER_PATTERNS

Plugin-specific option of ALWAYS_PARENT_FIRST_LOADER_PATTERNS. Plugins use this parent first list instead of the global version.

PLUGIN_ALWAYS_PARENT_FIRST_LOADER_PATTERNS_ADDITIONAL

public static final ConfigOption<List<String>> PLUGIN_ALWAYS_PARENT_FIRST_LOADER_PATTERNS_ADDITIONAL

FLINK_JVM_OPTIONS

public static final ConfigOption<String> FLINK_JVM_OPTIONS

FLINK_JM_JVM_OPTIONS

public static final ConfigOption<String> FLINK_JM_JVM_OPTIONS

FLINK_TM_JVM_OPTIONS

public static final ConfigOption<String> FLINK_TM_JVM_OPTIONS

FLINK_HS_JVM_OPTIONS

public static final ConfigOption<String> FLINK_HS_JVM_OPTIONS

FLINK_CLI_JVM_OPTIONS

public static final ConfigOption<String> FLINK_CLI_JVM_OPTIONS

FLINK_LOG_DIR

public static final ConfigOption<String> FLINK_LOG_DIR

This options is here only for documentation generation, it is only evaluated in the shell scripts.

FLINK_PID_DIR

public static final ConfigOption<String> FLINK_PID_DIR

The config parameter defining the directory for Flink PID file. see: bin/config.sh#KEY_ENV_PID_DIR and bin/config.sh#DEFAULT_ENV_PID_DIR

FLINK_LOG_MAX

public static final ConfigOption<Integer> FLINK_LOG_MAX

This options is here only for documentation generation, it is only evaluated in the shell scripts.

FLINK_SSH_OPTIONS

public static final ConfigOption<String> FLINK_SSH_OPTIONS

This options is here only for documentation generation, it is only evaluated in the shell scripts.

FLINK_HADOOP_CONF_DIR

public static final ConfigOption<String> FLINK_HADOOP_CONF_DIR

This options is here only for documentation generation, it is only evaluated in the shell scripts.

FLINK_YARN_CONF_DIR

public static final ConfigOption<String> FLINK_YARN_CONF_DIR

This options is here only for documentation generation, it is only evaluated in the shell scripts.

FLINK_HBASE_CONF_DIR

public static final ConfigOption<String> FLINK_HBASE_CONF_DIR

This options is here only for documentation generation, it is only evaluated in the shell scripts.

TMP_DIRS

public static final ConfigOption<String> TMP_DIRS

The config parameter defining the directories for temporary files, separated by ",", "|", or the system's File.pathSeparator.

DEFAULT_PARALLELISM

public static final ConfigOption<Integer> DEFAULT_PARALLELISM

DEFAULT_FILESYSTEM_SCHEME

public static final ConfigOption<String> DEFAULT_FILESYSTEM_SCHEME

The default filesystem scheme, used for paths that do not declare a scheme explicitly.

ALLOWED_FALLBACK_FILESYSTEMS

public static final ConfigOption<String> ALLOWED_FALLBACK_FILESYSTEMS

FILESYTEM_DEFAULT_OVERRIDE

public static final ConfigOption<Boolean> FILESYTEM_DEFAULT_OVERRIDE

Specifies whether file output writers should overwrite existing files by default.

FILESYSTEM_OUTPUT_ALWAYS_CREATE_DIRECTORY

public static final ConfigOption<Boolean> FILESYSTEM_OUTPUT_ALWAYS_CREATE_DIRECTORY

Specifies whether the file systems should always create a directory for the output, even with a parallelism of one.

Constructor Detail

CoreOptions

public CoreOptions()

Method Detail

getParentFirstLoaderPatterns

public static String[] getParentFirstLoaderPatterns(Configuration config)

getPluginParentFirstLoaderPatterns

public static String[] getPluginParentFirstLoaderPatterns(Configuration config)

mergeListsToArray

@Internal
public static String[] mergeListsToArray(List<String> base,
                                                   List<String> append)

fileSystemConnectionLimit

public static ConfigOption<Integer> fileSystemConnectionLimit(String scheme)

The total number of input plus output connections that a file system for the given scheme may open. Unlimited be default.

fileSystemConnectionLimitIn

public static ConfigOption<Integer> fileSystemConnectionLimitIn(String scheme)

The total number of input connections that a file system for the given scheme may open. Unlimited be default.

fileSystemConnectionLimitOut

public static ConfigOption<Integer> fileSystemConnectionLimitOut(String scheme)

The total number of output connections that a file system for the given scheme may open. Unlimited be default.

fileSystemConnectionLimitTimeout

public static ConfigOption<Long> fileSystemConnectionLimitTimeout(String scheme)

If any connection limit is configured, this option can be optionally set to define after which time (in milliseconds) stream opening fails with a timeout exception, if no stream connection becomes available. Unlimited timeout be default.

fileSystemConnectionLimitStreamInactivityTimeout

public static ConfigOption<Long> fileSystemConnectionLimitStreamInactivityTimeout(String scheme)

If any connection limit is configured, this option can be optionally set to define after which time (in milliseconds) inactive streams are reclaimed. This option can help to prevent that inactive streams make up the full pool of limited connections, and no further connections can be established. Unlimited timeout be default.