org.apache.flink.configuration

Class CoreOptions

java.lang.Object

org.apache.flink.configuration.CoreOptions

@PublicEvolving
public class CoreOptions
extends Object

The set of configuration options for core parameters.

Field Summary

Fields

Modifier and Type	Field and Description
static ConfigOption<String>	ALWAYS_PARENT_FIRST_LOADER_PATTERNS The namespace patterns for classes that are loaded with a preference from the parent classloader, meaning the application class path, rather than any user code jar file.
static ConfigOption<String>	ALWAYS_PARENT_FIRST_LOADER_PATTERNS_ADDITIONAL
static ConfigOption<String>	CLASSLOADER_RESOLVE_ORDER Defines the class resolution strategy when loading classes from user code, meaning whether to first check the user code jar ("child-first") or the application classpath ("parent-first")
static ConfigOption<String>	DEFAULT_FILESYSTEM_SCHEME The default filesystem scheme, used for paths that do not declare a scheme explicitly.
static ConfigOption<Integer>	DEFAULT_PARALLELISM
static ConfigOption<Boolean>	FILESYSTEM_OUTPUT_ALWAYS_CREATE_DIRECTORY Specifies whether the file systems should always create a directory for the output, even with a parallelism of one.
static ConfigOption<Boolean>	FILESYTEM_DEFAULT_OVERRIDE Specifies whether file output writers should overwrite existing files by default.
static ConfigOption<String>	FLINK_HADOOP_CONF_DIR This options is here only for documentation generation, it is only evaluated in the shell scripts.
static ConfigOption<String>	FLINK_JM_JVM_OPTIONS
static ConfigOption<String>	FLINK_JVM_OPTIONS
static ConfigOption<String>	FLINK_LOG_DIR This options is here only for documentation generation, it is only evaluated in the shell scripts.
static ConfigOption<Integer>	FLINK_LOG_MAX This options is here only for documentation generation, it is only evaluated in the shell scripts.
static ConfigOption<String>	FLINK_SSH_OPTIONS This options is here only for documentation generation, it is only evaluated in the shell scripts.
static ConfigOption<String>	FLINK_TM_JVM_OPTIONS
static ConfigOption<String>	FLINK_YARN_CONF_DIR This options is here only for documentation generation, it is only evaluated in the shell scripts.
static String	LEGACY_MODE Constant value for the old execution mode.
static ConfigOption<String>	MODE Switch to select the execution mode.
static String	NEW_MODE Constant value for the new execution mode.
static ConfigOption<String>	TMP_DIRS The config parameter defining the directories for temporary files, separated by ",", "|", or the system's File.pathSeparator.

Constructor Summary

Constructors

Constructor and Description
CoreOptions()

Method Summary

Modifier and Type	Method and Description
static ConfigOption<Integer>	fileSystemConnectionLimit(String scheme) The total number of input plus output connections that a file system for the given scheme may open.
static ConfigOption<Integer>	fileSystemConnectionLimitIn(String scheme) The total number of input connections that a file system for the given scheme may open.
static ConfigOption<Integer>	fileSystemConnectionLimitOut(String scheme) The total number of output connections that a file system for the given scheme may open.
static ConfigOption<Long>	fileSystemConnectionLimitStreamInactivityTimeout(String scheme) If any connection limit is configured, this option can be optionally set to define after which time (in milliseconds) inactive streams are reclaimed.
static ConfigOption<Long>	fileSystemConnectionLimitTimeout(String scheme) If any connection limit is configured, this option can be optionally set to define after which time (in milliseconds) stream opening fails with a timeout exception, if no stream connection becomes available.
static String[]	getParentFirstLoaderPatterns(Configuration config)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CLASSLOADER_RESOLVE_ORDER

public static final ConfigOption<String> CLASSLOADER_RESOLVE_ORDER

Defines the class resolution strategy when loading classes from user code, meaning whether to first check the user code jar ("child-first") or the application classpath ("parent-first")

The default settings indicate to load classes first from the user code jar, which means that user code jars can include and load different dependencies than Flink uses (transitively).

Exceptions to the rules are defined via ALWAYS_PARENT_FIRST_LOADER_PATTERNS.

ALWAYS_PARENT_FIRST_LOADER_PATTERNS

public static final ConfigOption<String> ALWAYS_PARENT_FIRST_LOADER_PATTERNS

The namespace patterns for classes that are loaded with a preference from the parent classloader, meaning the application class path, rather than any user code jar file. This option only has an effect when CLASSLOADER_RESOLVE_ORDER is set to "child-first".

It is important that all classes whose objects move between Flink's runtime and any user code (including Flink connectors that run as part of the user code) are covered by these patterns here. Otherwise it is be possible that the Flink runtime and the user code load two different copies of a class through the different class loaders. That leads to errors like "X cannot be cast to X" exceptions, where both class names are equal, or "X cannot be assigned to Y", where X should be a subclass of Y.

The following classes are loaded parent-first, to avoid any duplication:

• All core Java classes (java.*), because they must never be duplicated.
• All core Scala classes (scala.*). Currently Scala is used in the Flink runtime and in the user code, and some Scala classes cross the boundary, such as the FunctionX classes. That may change if Scala eventually lives purely as part of the user code.
• All Flink classes (org.apache.flink.*). Note that this means that connectors and formats (flink-avro, etc) are loaded parent-first as well if they are in the core classpath.
• Java annotations and loggers, defined by the following list: javax.annotation;org.slf4j;org.apache.log4j;org.apache.logging;org.apache.commons.logging;ch.qos.logback. This is done for convenience, to avoid duplication of annotations and multiple log bindings.

ALWAYS_PARENT_FIRST_LOADER_PATTERNS_ADDITIONAL

public static final ConfigOption<String> ALWAYS_PARENT_FIRST_LOADER_PATTERNS_ADDITIONAL

FLINK_JVM_OPTIONS

public static final ConfigOption<String> FLINK_JVM_OPTIONS

FLINK_JM_JVM_OPTIONS

public static final ConfigOption<String> FLINK_JM_JVM_OPTIONS

FLINK_TM_JVM_OPTIONS

public static final ConfigOption<String> FLINK_TM_JVM_OPTIONS

FLINK_LOG_DIR

public static final ConfigOption<String> FLINK_LOG_DIR

This options is here only for documentation generation, it is only evaluated in the shell scripts.

FLINK_LOG_MAX

public static final ConfigOption<Integer> FLINK_LOG_MAX

This options is here only for documentation generation, it is only evaluated in the shell scripts.

FLINK_SSH_OPTIONS

public static final ConfigOption<String> FLINK_SSH_OPTIONS

This options is here only for documentation generation, it is only evaluated in the shell scripts.

FLINK_HADOOP_CONF_DIR

public static final ConfigOption<String> FLINK_HADOOP_CONF_DIR

This options is here only for documentation generation, it is only evaluated in the shell scripts.

FLINK_YARN_CONF_DIR

public static final ConfigOption<String> FLINK_YARN_CONF_DIR

This options is here only for documentation generation, it is only evaluated in the shell scripts.

TMP_DIRS

public static final ConfigOption<String> TMP_DIRS

The config parameter defining the directories for temporary files, separated by ",", "|", or the system's File.pathSeparator.

DEFAULT_PARALLELISM

public static final ConfigOption<Integer> DEFAULT_PARALLELISM

DEFAULT_FILESYSTEM_SCHEME

public static final ConfigOption<String> DEFAULT_FILESYSTEM_SCHEME

The default filesystem scheme, used for paths that do not declare a scheme explicitly.

FILESYTEM_DEFAULT_OVERRIDE

public static final ConfigOption<Boolean> FILESYTEM_DEFAULT_OVERRIDE

Specifies whether file output writers should overwrite existing files by default.

FILESYSTEM_OUTPUT_ALWAYS_CREATE_DIRECTORY

public static final ConfigOption<Boolean> FILESYSTEM_OUTPUT_ALWAYS_CREATE_DIRECTORY

Specifies whether the file systems should always create a directory for the output, even with a parallelism of one.

NEW_MODE

public static final String NEW_MODE

Constant value for the new execution mode.

See Also:

Constant Field Values

LEGACY_MODE

public static final String LEGACY_MODE

Constant value for the old execution mode.

See Also:

Constant Field Values

MODE

public static final ConfigOption<String> MODE

Switch to select the execution mode. Possible values are NEW_MODE and LEGACY_MODE.

Constructor Detail

CoreOptions

public CoreOptions()

Method Detail

getParentFirstLoaderPatterns

public static String[] getParentFirstLoaderPatterns(Configuration config)

fileSystemConnectionLimit

public static ConfigOption<Integer> fileSystemConnectionLimit(String scheme)

The total number of input plus output connections that a file system for the given scheme may open. Unlimited be default.

fileSystemConnectionLimitIn

public static ConfigOption<Integer> fileSystemConnectionLimitIn(String scheme)

The total number of input connections that a file system for the given scheme may open. Unlimited be default.

fileSystemConnectionLimitOut

public static ConfigOption<Integer> fileSystemConnectionLimitOut(String scheme)

The total number of output connections that a file system for the given scheme may open. Unlimited be default.

fileSystemConnectionLimitTimeout

public static ConfigOption<Long> fileSystemConnectionLimitTimeout(String scheme)

If any connection limit is configured, this option can be optionally set to define after which time (in milliseconds) stream opening fails with a timeout exception, if no stream connection becomes available. Unlimited timeout be default.

fileSystemConnectionLimitStreamInactivityTimeout

public static ConfigOption<Long> fileSystemConnectionLimitStreamInactivityTimeout(String scheme)

If any connection limit is configured, this option can be optionally set to define after which time (in milliseconds) inactive streams are reclaimed. This option can help to prevent that inactive streams make up the full pool of limited connections, and no further connections can be established. Unlimited timeout be default.