org.apache.flink.configuration

Class NettyShuffleEnvironmentOptions

java.lang.Object

org.apache.flink.configuration.NettyShuffleEnvironmentOptions

@PublicEvolving
public class NettyShuffleEnvironmentOptions
extends Object

The set of configuration options relating to network stack.

Field Summary

Fields

Modifier and Type	Field and Description
static ConfigOption<Boolean>	BATCH_SHUFFLE_COMPRESSION_ENABLED Boolean flag indicating whether the shuffle data will be compressed for batch shuffle mode.
static ConfigOption<Integer>	CLIENT_CONNECT_TIMEOUT_SECONDS
static ConfigOption<Integer>	CLIENT_TCP_KEEP_COUNT
static ConfigOption<Integer>	CLIENT_TCP_KEEP_IDLE_SECONDS
static ConfigOption<Integer>	CLIENT_TCP_KEEP_INTERVAL_SECONDS
static ConfigOption<Integer>	CONNECT_BACKLOG
static ConfigOption<Integer>	DATA_BIND_PORT The local network port that the task manager listen at for data exchange.
static ConfigOption<Integer>	DATA_PORT The default network port the task manager expects to receive transfer envelopes on.
static ConfigOption<Boolean>	DATA_SSL_ENABLED Config parameter to override SSL support for taskmanager's data transport.
static ConfigOption<Long>	HYBRID_SHUFFLE_NUM_RETAINED_IN_MEMORY_REGIONS_MAX Max number of hybrid retained regions in memory.
static ConfigOption<Integer>	HYBRID_SHUFFLE_SPILLED_INDEX_REGION_GROUP_SIZE Region group size of hybrid spilled file data index.
static ConfigOption<Integer>	MAX_NUM_TCP_CONNECTIONS The maximum number of tpc connections between taskmanagers for data communication.
static ConfigOption<String>	NETWORK_BLOCKING_SHUFFLE_TYPE
static ConfigOption<Float>	NETWORK_BUFFERS_MEMORY_FRACTION Deprecated. use TaskManagerOptions.NETWORK_MEMORY_FRACTION instead
static ConfigOption<String>	NETWORK_BUFFERS_MEMORY_MAX Deprecated. use TaskManagerOptions.NETWORK_MEMORY_MAX instead
static ConfigOption<String>	NETWORK_BUFFERS_MEMORY_MIN Deprecated. use TaskManagerOptions.NETWORK_MEMORY_MIN instead
static ConfigOption<Integer>	NETWORK_BUFFERS_PER_CHANNEL Number of network buffers for each outgoing/incoming channel (subpartition/input channel).
static ConfigOption<Boolean>	NETWORK_DETAILED_METRICS Boolean flag to enable/disable more detailed metrics about inbound/outbound network queue lengths.
static ConfigOption<Long>	NETWORK_EXCLUSIVE_BUFFERS_REQUEST_TIMEOUT_MILLISECONDS The timeout for requesting exclusive buffers for each channel.
static ConfigOption<Integer>	NETWORK_EXTRA_BUFFERS_PER_GATE Number of floating network buffers for each outgoing/incoming gate (result partition/input gate).
static ConfigOption<Boolean>	NETWORK_HYBRID_SHUFFLE_ENABLE_NEW_MODE The option to enable the new mode of hybrid shuffle.
static ConfigOption<String>	NETWORK_HYBRID_SHUFFLE_REMOTE_STORAGE_BASE_PATH The option to configure the base remote storage path for hybrid shuffle.
static ConfigOption<Integer>	NETWORK_MAX_BUFFERS_PER_CHANNEL Number of max buffers can be used for each output subpartition.
static ConfigOption<Integer>	NETWORK_MAX_OVERDRAFT_BUFFERS_PER_GATE Number of max overdraft network buffers to use for each ResultPartition.
static ConfigOption<Integer>	NETWORK_NUM_BUFFERS Deprecated. use TaskManagerOptions.NETWORK_MEMORY_FRACTION, TaskManagerOptions.NETWORK_MEMORY_MIN, and TaskManagerOptions.NETWORK_MEMORY_MAX instead
static ConfigOption<Integer>	NETWORK_READ_MAX_REQUIRED_BUFFERS_PER_GATE The maximum number of network read buffers that are required by an input gate.
static ConfigOption<Integer>	NETWORK_REQUEST_BACKOFF_INITIAL Minimum backoff for partition requests of input channels.
static ConfigOption<Integer>	NETWORK_REQUEST_BACKOFF_MAX Maximum backoff for partition requests of input channels.
static ConfigOption<Integer>	NETWORK_RETRIES
static ConfigOption<Integer>	NETWORK_SORT_SHUFFLE_MIN_BUFFERS Minimum number of network buffers required per blocking result partition for sort-shuffle.
static ConfigOption<Integer>	NETWORK_SORT_SHUFFLE_MIN_PARALLELISM Parallelism threshold to switch between sort-based blocking shuffle and hash-based blocking shuffle.
static ConfigOption<Integer>	NUM_ARENAS
static ConfigOption<Integer>	NUM_THREADS_CLIENT
static ConfigOption<Integer>	NUM_THREADS_SERVER
static ConfigOption<Integer>	SEND_RECEIVE_BUFFER_SIZE
static ConfigOption<String>	SHUFFLE_COMPRESSION_CODEC The codec to be used when compressing shuffle data.
static ConfigOption<Boolean>	TCP_CONNECTION_REUSE_ACROSS_JOBS_ENABLED Whether to reuse tcp connections across multi jobs.
static ConfigOption<String>	TRANSPORT_TYPE

Method Summary

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DATA_PORT

public static final ConfigOption<Integer> DATA_PORT

The default network port the task manager expects to receive transfer envelopes on. The 0 means that the TaskManager searches for a free port.

DATA_BIND_PORT

public static final ConfigOption<Integer> DATA_BIND_PORT

The local network port that the task manager listen at for data exchange.

DATA_SSL_ENABLED

public static final ConfigOption<Boolean> DATA_SSL_ENABLED

Config parameter to override SSL support for taskmanager's data transport.

BATCH_SHUFFLE_COMPRESSION_ENABLED

public static final ConfigOption<Boolean> BATCH_SHUFFLE_COMPRESSION_ENABLED

Boolean flag indicating whether the shuffle data will be compressed for batch shuffle mode.

Note: Data is compressed per buffer and compression can incur extra CPU overhead so it is more effective for IO bounded scenario when data compression ratio is high.

SHUFFLE_COMPRESSION_CODEC

@Experimental
public static final ConfigOption<String> SHUFFLE_COMPRESSION_CODEC

The codec to be used when compressing shuffle data.

NETWORK_DETAILED_METRICS

public static final ConfigOption<Boolean> NETWORK_DETAILED_METRICS

Boolean flag to enable/disable more detailed metrics about inbound/outbound network queue lengths.

NETWORK_NUM_BUFFERS

@Deprecated
public static final ConfigOption<Integer> NETWORK_NUM_BUFFERS

Deprecated. use TaskManagerOptions.NETWORK_MEMORY_FRACTION, TaskManagerOptions.NETWORK_MEMORY_MIN, and TaskManagerOptions.NETWORK_MEMORY_MAX instead

Number of buffers used in the network stack. This defines the number of possible tasks and shuffles.

NETWORK_BUFFERS_MEMORY_FRACTION

@Deprecated
public static final ConfigOption<Float> NETWORK_BUFFERS_MEMORY_FRACTION

Deprecated. use TaskManagerOptions.NETWORK_MEMORY_FRACTION instead

Fraction of JVM memory to use for network buffers.

NETWORK_BUFFERS_MEMORY_MIN

@Deprecated
public static final ConfigOption<String> NETWORK_BUFFERS_MEMORY_MIN

Deprecated. use TaskManagerOptions.NETWORK_MEMORY_MIN instead

Minimum memory size for network buffers.

NETWORK_BUFFERS_MEMORY_MAX

@Deprecated
public static final ConfigOption<String> NETWORK_BUFFERS_MEMORY_MAX

Deprecated. use TaskManagerOptions.NETWORK_MEMORY_MAX instead

Maximum memory size for network buffers.

MAX_NUM_TCP_CONNECTIONS

public static final ConfigOption<Integer> MAX_NUM_TCP_CONNECTIONS

The maximum number of tpc connections between taskmanagers for data communication.

NETWORK_READ_MAX_REQUIRED_BUFFERS_PER_GATE

@Experimental
public static final ConfigOption<Integer> NETWORK_READ_MAX_REQUIRED_BUFFERS_PER_GATE

The maximum number of network read buffers that are required by an input gate. (An input gate is responsible for reading data from all subtasks of an upstream task.) The number of buffers needed by an input gate is dynamically calculated in runtime, depending on various factors (e.g., the parallelism of the upstream task). Among the calculated number of needed buffers, the part below this configured value is required, while the excess part, if any, is optional. A task will fail if the required buffers cannot be obtained in runtime. A task will not fail due to not obtaining optional buffers, but may suffer a performance reduction.

NETWORK_BUFFERS_PER_CHANNEL

public static final ConfigOption<Integer> NETWORK_BUFFERS_PER_CHANNEL

Number of network buffers for each outgoing/incoming channel (subpartition/input channel). The minimum valid value for the option is 0. When the option is configured as 0, the exclusive network buffers used per downstream incoming channel will be 0, but for each upstream outgoing channel, max(1, configured value) will be used. In other words we ensure that, for performance reasons, at least one buffer is used per outgoing channel regardless of the configuration.

NETWORK_EXTRA_BUFFERS_PER_GATE

public static final ConfigOption<Integer> NETWORK_EXTRA_BUFFERS_PER_GATE

Number of floating network buffers for each outgoing/incoming gate (result partition/input gate).

NETWORK_SORT_SHUFFLE_MIN_BUFFERS

public static final ConfigOption<Integer> NETWORK_SORT_SHUFFLE_MIN_BUFFERS

Minimum number of network buffers required per blocking result partition for sort-shuffle.

NETWORK_SORT_SHUFFLE_MIN_PARALLELISM

public static final ConfigOption<Integer> NETWORK_SORT_SHUFFLE_MIN_PARALLELISM

Parallelism threshold to switch between sort-based blocking shuffle and hash-based blocking shuffle.

HYBRID_SHUFFLE_SPILLED_INDEX_REGION_GROUP_SIZE

public static final ConfigOption<Integer> HYBRID_SHUFFLE_SPILLED_INDEX_REGION_GROUP_SIZE

Region group size of hybrid spilled file data index.

HYBRID_SHUFFLE_NUM_RETAINED_IN_MEMORY_REGIONS_MAX

public static final ConfigOption<Long> HYBRID_SHUFFLE_NUM_RETAINED_IN_MEMORY_REGIONS_MAX

Max number of hybrid retained regions in memory.

NETWORK_MAX_BUFFERS_PER_CHANNEL

public static final ConfigOption<Integer> NETWORK_MAX_BUFFERS_PER_CHANNEL

Number of max buffers can be used for each output subpartition.

NETWORK_MAX_OVERDRAFT_BUFFERS_PER_GATE

public static final ConfigOption<Integer> NETWORK_MAX_OVERDRAFT_BUFFERS_PER_GATE

Number of max overdraft network buffers to use for each ResultPartition.

NETWORK_EXCLUSIVE_BUFFERS_REQUEST_TIMEOUT_MILLISECONDS

public static final ConfigOption<Long> NETWORK_EXCLUSIVE_BUFFERS_REQUEST_TIMEOUT_MILLISECONDS

The timeout for requesting exclusive buffers for each channel.

NETWORK_HYBRID_SHUFFLE_ENABLE_NEW_MODE

@Experimental
public static final ConfigOption<Boolean> NETWORK_HYBRID_SHUFFLE_ENABLE_NEW_MODE

The option to enable the new mode of hybrid shuffle.

NETWORK_HYBRID_SHUFFLE_REMOTE_STORAGE_BASE_PATH

@Experimental
public static final ConfigOption<String> NETWORK_HYBRID_SHUFFLE_REMOTE_STORAGE_BASE_PATH

The option to configure the base remote storage path for hybrid shuffle.

NETWORK_BLOCKING_SHUFFLE_TYPE

public static final ConfigOption<String> NETWORK_BLOCKING_SHUFFLE_TYPE

TCP_CONNECTION_REUSE_ACROSS_JOBS_ENABLED

public static final ConfigOption<Boolean> TCP_CONNECTION_REUSE_ACROSS_JOBS_ENABLED

Whether to reuse tcp connections across multi jobs. If set to true, tcp connections will not be released after job finishes. The subsequent jobs will be free from the overhead of the connection re-establish. However, this may lead to an increase in the total number of connections on your machine. When it reaches the upper limit, you can set it to false to release idle connections.

Note: To avoid connection leak, you must set MAX_NUM_TCP_CONNECTIONS to a smaller value before you enable tcp connection reuse.

NUM_ARENAS

public static final ConfigOption<Integer> NUM_ARENAS

NUM_THREADS_SERVER

public static final ConfigOption<Integer> NUM_THREADS_SERVER

NUM_THREADS_CLIENT

public static final ConfigOption<Integer> NUM_THREADS_CLIENT

CONNECT_BACKLOG

public static final ConfigOption<Integer> CONNECT_BACKLOG

CLIENT_CONNECT_TIMEOUT_SECONDS

public static final ConfigOption<Integer> CLIENT_CONNECT_TIMEOUT_SECONDS

NETWORK_RETRIES

public static final ConfigOption<Integer> NETWORK_RETRIES

SEND_RECEIVE_BUFFER_SIZE

public static final ConfigOption<Integer> SEND_RECEIVE_BUFFER_SIZE

TRANSPORT_TYPE

public static final ConfigOption<String> TRANSPORT_TYPE

CLIENT_TCP_KEEP_IDLE_SECONDS

public static final ConfigOption<Integer> CLIENT_TCP_KEEP_IDLE_SECONDS

CLIENT_TCP_KEEP_INTERVAL_SECONDS

public static final ConfigOption<Integer> CLIENT_TCP_KEEP_INTERVAL_SECONDS

CLIENT_TCP_KEEP_COUNT

public static final ConfigOption<Integer> CLIENT_TCP_KEEP_COUNT

NETWORK_REQUEST_BACKOFF_INITIAL

public static final ConfigOption<Integer> NETWORK_REQUEST_BACKOFF_INITIAL

Minimum backoff for partition requests of input channels.

NETWORK_REQUEST_BACKOFF_MAX

public static final ConfigOption<Integer> NETWORK_REQUEST_BACKOFF_MAX

Maximum backoff for partition requests of input channels.