org.apache.flink.yarn.configuration

Class YarnConfigOptions

java.lang.Object

org.apache.flink.yarn.configuration.YarnConfigOptions

public class YarnConfigOptions
extends Object

This class holds configuration constants used by Flink's YARN runners.

These options are not expected to be ever configured by users explicitly.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	YarnConfigOptions.UserJarInclusion

Field Summary

Fields

Modifier and Type	Field and Description
static ConfigOption<Integer>	APP_MASTER_VCORES The vcores used by YARN application master.
static ConfigOption<Long>	APPLICATION_ATTEMPT_FAILURE_VALIDITY_INTERVAL The config parameter defining the attemptFailuresValidityInterval of Yarn application.
static ConfigOption<String>	APPLICATION_ATTEMPTS Set the number of retries for failed YARN ApplicationMasters/JobManagers in high availability mode.
static ConfigOption<String>	APPLICATION_ID
static ConfigOption<String>	APPLICATION_MASTER_PORT The config parameter defining the Akka actor system port for the ApplicationMaster and JobManager.
static ConfigOption<String>	APPLICATION_NAME
static ConfigOption<Integer>	APPLICATION_PRIORITY A non-negative integer indicating the priority for submitting a Flink YARN application.
static ConfigOption<String>	APPLICATION_QUEUE
static ConfigOption<String>	APPLICATION_TAGS A comma-separated list of strings to use as YARN application tags.
static ConfigOption<String>	APPLICATION_TYPE
static ConfigOption<YarnConfigOptions.UserJarInclusion>	CLASSPATH_INCLUDE_USER_JAR Defines whether user-jars are included in the system class path for per-job-clusters as well as their positioning in the path.
static ConfigOption<Integer>	CONTAINER_REQUEST_HEARTBEAT_INTERVAL_MILLISECONDS The heartbeat interval between the Application Master and the YARN Resource Manager if Flink is requesting containers.
static ConfigOption<String>	EXTERNAL_RESOURCE_YARN_CONFIG_KEY If configured, Flink will add this key to the resource profile of container request to Yarn.
static String	EXTERNAL_RESOURCE_YARN_CONFIG_KEY_SUFFIX Defines the configuration key of that external resource in Yarn.
static ConfigOption<Integer>	FILE_REPLICATION Yarn session client uploads flink jar and user libs to file system (hdfs/s3) as local resource for yarn application context.
static ConfigOption<String>	FLINK_DIST_JAR
static ConfigOption<String>	HADOOP_CONFIG_KEY
static ConfigOption<Integer>	HEARTBEAT_DELAY_SECONDS The heartbeat interval between the Application Master and the YARN Resource Manager.
static ConfigOption<String>	LOCALIZED_KEYTAB_PATH
static ConfigOption<String>	NODE_LABEL
static ConfigOption<String>	PROPERTIES_FILE_LOCATION When a Flink job is submitted to YARN, the JobManager's host and the number of available processing slots is written into a properties file, so that the Flink client is able to pick those details up.
static ConfigOption<List<String>>	PROVIDED_LIB_DIRS
static ConfigOption<List<String>>	SHIP_ARCHIVES
static ConfigOption<List<String>>	SHIP_FILES
static ConfigOption<Boolean>	SHIP_LOCAL_KEYTAB
static ConfigOption<String>	STAGING_DIRECTORY
static ConfigOption<Integer>	VCORES The vcores exposed by YARN.
static ConfigOption<List<String>>	YARN_ACCESS
static ConfigOption<String>	YARN_CONFIG_KEY

Method Summary

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

APP_MASTER_VCORES

public static final ConfigOption<Integer> APP_MASTER_VCORES

The vcores used by YARN application master.

CLASSPATH_INCLUDE_USER_JAR

public static final ConfigOption<YarnConfigOptions.UserJarInclusion> CLASSPATH_INCLUDE_USER_JAR

Defines whether user-jars are included in the system class path for per-job-clusters as well as their positioning in the path. They can be positioned at the beginning (FIRST), at the end (LAST), or be positioned based on their name (ORDER). DISABLED means the user-jars are excluded from the system class path.

VCORES

public static final ConfigOption<Integer> VCORES

The vcores exposed by YARN.

APPLICATION_ATTEMPTS

public static final ConfigOption<String> APPLICATION_ATTEMPTS

Set the number of retries for failed YARN ApplicationMasters/JobManagers in high availability mode. This value is usually limited by YARN. By default, it's 1 in the standalone case and 2 in the high availability case.

>Note: This option returns a String since Integer options must have a static default value.

APPLICATION_ATTEMPT_FAILURE_VALIDITY_INTERVAL

public static final ConfigOption<Long> APPLICATION_ATTEMPT_FAILURE_VALIDITY_INTERVAL

The config parameter defining the attemptFailuresValidityInterval of Yarn application.

HEARTBEAT_DELAY_SECONDS

public static final ConfigOption<Integer> HEARTBEAT_DELAY_SECONDS

The heartbeat interval between the Application Master and the YARN Resource Manager.

CONTAINER_REQUEST_HEARTBEAT_INTERVAL_MILLISECONDS

public static final ConfigOption<Integer> CONTAINER_REQUEST_HEARTBEAT_INTERVAL_MILLISECONDS

The heartbeat interval between the Application Master and the YARN Resource Manager if Flink is requesting containers.

PROPERTIES_FILE_LOCATION

public static final ConfigOption<String> PROPERTIES_FILE_LOCATION

When a Flink job is submitted to YARN, the JobManager's host and the number of available processing slots is written into a properties file, so that the Flink client is able to pick those details up. This configuration parameter allows changing the default location of that file (for example for environments sharing a Flink installation between users)

APPLICATION_MASTER_PORT

public static final ConfigOption<String> APPLICATION_MASTER_PORT

The config parameter defining the Akka actor system port for the ApplicationMaster and JobManager. The port can either be a port, such as "9123", a range of ports: "50100-50200" or a list of ranges and or points: "50100-50200,50300-50400,51234". Setting the port to 0 will let the OS choose an available port.

APPLICATION_PRIORITY

public static final ConfigOption<Integer> APPLICATION_PRIORITY

A non-negative integer indicating the priority for submitting a Flink YARN application. It will only take effect if YARN priority scheduling setting is enabled. Larger integer corresponds with higher priority. If priority is negative or set to '-1'(default), Flink will unset yarn priority setting and use cluster default priority.

See Also:

YARN Capacity Scheduling Doc

FILE_REPLICATION

public static final ConfigOption<Integer> FILE_REPLICATION

Yarn session client uploads flink jar and user libs to file system (hdfs/s3) as local resource for yarn application context. The replication number changes the how many replica of each of these files in hdfs/s3. It is useful to accelerate this container bootstrap, when a Flink application needs more than one hundred of containers. If it is not configured, Flink will use the default replication value in hadoop configuration.

APPLICATION_TAGS

public static final ConfigOption<String> APPLICATION_TAGS

A comma-separated list of strings to use as YARN application tags.

STAGING_DIRECTORY

public static final ConfigOption<String> STAGING_DIRECTORY

SHIP_FILES

public static final ConfigOption<List<String>> SHIP_FILES

SHIP_ARCHIVES

public static final ConfigOption<List<String>> SHIP_ARCHIVES

FLINK_DIST_JAR

public static final ConfigOption<String> FLINK_DIST_JAR

APPLICATION_ID

public static final ConfigOption<String> APPLICATION_ID

APPLICATION_QUEUE

public static final ConfigOption<String> APPLICATION_QUEUE

APPLICATION_NAME

public static final ConfigOption<String> APPLICATION_NAME

APPLICATION_TYPE

public static final ConfigOption<String> APPLICATION_TYPE

NODE_LABEL

public static final ConfigOption<String> NODE_LABEL

SHIP_LOCAL_KEYTAB

public static final ConfigOption<Boolean> SHIP_LOCAL_KEYTAB

LOCALIZED_KEYTAB_PATH

public static final ConfigOption<String> LOCALIZED_KEYTAB_PATH

PROVIDED_LIB_DIRS

public static final ConfigOption<List<String>> PROVIDED_LIB_DIRS

YARN_ACCESS

public static final ConfigOption<List<String>> YARN_ACCESS

HADOOP_CONFIG_KEY

public static final ConfigOption<String> HADOOP_CONFIG_KEY

YARN_CONFIG_KEY

public static final ConfigOption<String> YARN_CONFIG_KEY

EXTERNAL_RESOURCE_YARN_CONFIG_KEY_SUFFIX

public static final String EXTERNAL_RESOURCE_YARN_CONFIG_KEY_SUFFIX

Defines the configuration key of that external resource in Yarn. This is used as a suffix in an actual config.

See Also:

Constant Field Values

EXTERNAL_RESOURCE_YARN_CONFIG_KEY

public static final ConfigOption<String> EXTERNAL_RESOURCE_YARN_CONFIG_KEY

If configured, Flink will add this key to the resource profile of container request to Yarn. The value will be set to ExternalResourceOptions.EXTERNAL_RESOURCE_AMOUNT.

It is intentionally included into user docs while unused.