@PublicEvolving public class NettyShuffleEnvironmentOptions extends Object
Modifier and Type | Class and Description |
---|---|
static class |
NettyShuffleEnvironmentOptions.CompressionCodec
Supported compression codec.
|
Modifier and Type | Field and Description |
---|---|
static ConfigOption<Boolean> |
BATCH_SHUFFLE_COMPRESSION_ENABLED
Deprecated.
This option is deprecated in 1.20 and will be removed in 2.0. Please set the
SHUFFLE_COMPRESSION_CODEC to NONE to disable the
compression. |
static ConfigOption<Integer> |
CLIENT_CONNECT_TIMEOUT_SECONDS |
static ConfigOption<Integer> |
CLIENT_TCP_KEEP_COUNT |
static ConfigOption<Integer> |
CLIENT_TCP_KEEP_IDLE_SECONDS |
static ConfigOption<Integer> |
CLIENT_TCP_KEEP_INTERVAL_SECONDS |
static ConfigOption<Integer> |
CONNECT_BACKLOG
Deprecated.
It is not recommended to use the option. It is deprecated in 1.20 and will be
totally removed in 2.0.
|
static ConfigOption<String> |
DATA_BIND_PORT
The local network port that the task manager listen at for data exchange.
|
static ConfigOption<Integer> |
DATA_PORT
The default network port the task manager expects to receive transfer envelopes on.
|
static ConfigOption<Boolean> |
DATA_SSL_ENABLED
Config parameter to override SSL support for taskmanager's data transport.
|
static ConfigOption<Long> |
HYBRID_SHUFFLE_NUM_RETAINED_IN_MEMORY_REGIONS_MAX
Deprecated.
The option is only used in the legacy hybrid shuffle mode. It is deprecated in
1.20 and will be totally removed in 2.0, as the legacy hybrid shuffle mode will be
removed in 2.0.
|
static ConfigOption<Integer> |
HYBRID_SHUFFLE_SPILLED_INDEX_REGION_GROUP_SIZE
Deprecated.
The option is only used in the legacy hybrid shuffle mode. It is deprecated in
1.20 and will be totally removed in 2.0, as the legacy hybrid shuffle mode will be
removed in 2.0.
|
static ConfigOption<Integer> |
MAX_NUM_TCP_CONNECTIONS
Deprecated.
The option is unnecessary. It is deprecated in 1.20 and will be removed and
hard-coded to 1 in 2.0.
|
static ConfigOption<String> |
NETWORK_BLOCKING_SHUFFLE_TYPE
Deprecated.
The hash-based blocking shuffle is deprecated in 1.20 and will be totally removed
in 2.0.
|
static ConfigOption<Float> |
NETWORK_BUFFERS_MEMORY_FRACTION
Deprecated.
use
TaskManagerOptions.NETWORK_MEMORY_FRACTION instead |
static ConfigOption<String> |
NETWORK_BUFFERS_MEMORY_MAX
Deprecated.
use
TaskManagerOptions.NETWORK_MEMORY_MAX instead |
static ConfigOption<String> |
NETWORK_BUFFERS_MEMORY_MIN
Deprecated.
use
TaskManagerOptions.NETWORK_MEMORY_MIN instead |
static ConfigOption<Integer> |
NETWORK_BUFFERS_PER_CHANNEL
Deprecated.
This option is deprecated in 1.20 and will be removed in 2.0 to simplify the
configuration of network buffers.
|
static ConfigOption<java.time.Duration> |
NETWORK_BUFFERS_REQUEST_TIMEOUT
The timeout for requesting buffers for each channel.
|
static ConfigOption<Boolean> |
NETWORK_DETAILED_METRICS
Boolean flag to enable/disable more detailed metrics about inbound/outbound network queue
lengths.
|
static ConfigOption<Long> |
NETWORK_EXCLUSIVE_BUFFERS_REQUEST_TIMEOUT_MILLISECONDS
Deprecated.
This option is deprecated in 1.20 and will be removed in 2.0 to simplify the
configuration of network buffers. Please use
NETWORK_BUFFERS_REQUEST_TIMEOUT instead. |
static ConfigOption<Integer> |
NETWORK_EXTRA_BUFFERS_PER_GATE
Deprecated.
This option is deprecated in 1.20 and will be removed in 2.0 to simplify the
configuration of network buffers.
|
static ConfigOption<Boolean> |
NETWORK_HYBRID_SHUFFLE_ENABLE_NEW_MODE
Deprecated.
This option is deprecated in 1.20 and will be totally removed in 2.0, as the
legacy hybrid shuffle mode will be removed in 2.0.
|
static ConfigOption<String> |
NETWORK_HYBRID_SHUFFLE_EXTERNAL_REMOTE_TIER_FACTORY_CLASS_NAME
The option to configure the tiered factory creator remote class name for hybrid shuffle.
|
static ConfigOption<String> |
NETWORK_HYBRID_SHUFFLE_REMOTE_STORAGE_BASE_PATH
The option to configure the base remote storage path for hybrid shuffle.
|
static ConfigOption<Integer> |
NETWORK_MAX_BUFFERS_PER_CHANNEL
Deprecated.
This option is deprecated in 1.20 and will be removed in 2.0 to simplify the
configuration of network buffers.
|
static ConfigOption<Integer> |
NETWORK_MAX_OVERDRAFT_BUFFERS_PER_GATE
Deprecated.
This option is deprecated in 1.20 and will be removed in 2.0 to simplify the
configuration of network buffers.
|
static ConfigOption<Integer> |
NETWORK_NUM_BUFFERS
Deprecated.
|
static ConfigOption<java.time.Duration> |
NETWORK_PARTITION_REQUEST_TIMEOUT
The timeout for partition request listener in result partition manager.
|
static ConfigOption<Integer> |
NETWORK_READ_MAX_REQUIRED_BUFFERS_PER_GATE
The maximum number of network read buffers that are required by an input gate.
|
static ConfigOption<Integer> |
NETWORK_REQUEST_BACKOFF_INITIAL
Minimum backoff for partition requests of input channels.
|
static ConfigOption<Integer> |
NETWORK_REQUEST_BACKOFF_MAX
Maximum backoff for partition requests of input channels.
|
static ConfigOption<Integer> |
NETWORK_RETRIES |
static ConfigOption<Integer> |
NETWORK_SORT_SHUFFLE_MIN_BUFFERS
Minimum number of network buffers required per blocking result partition for sort-shuffle.
|
static ConfigOption<Integer> |
NETWORK_SORT_SHUFFLE_MIN_PARALLELISM
Deprecated.
The hash-based blocking shuffle is deprecated in 1.20 and will be totally removed
in 2.0.
|
static ConfigOption<Integer> |
NUM_ARENAS
Deprecated.
It is not recommended to use the option. It is deprecated in 1.20 and will be
totally removed in 2.0.
|
static ConfigOption<Integer> |
NUM_THREADS_CLIENT
Deprecated.
It is not recommended to use the option. It is deprecated in 1.20 and will be
totally removed in 2.0.
|
static ConfigOption<Integer> |
NUM_THREADS_SERVER
Deprecated.
It is not recommended to use the option. It is deprecated in 1.20 and will be
totally removed in 2.0.
|
static ConfigOption<Integer> |
SEND_RECEIVE_BUFFER_SIZE
Deprecated.
It is not recommended to use the option. It is deprecated in 1.20 and will be
totally removed in 2.0.
|
static ConfigOption<NettyShuffleEnvironmentOptions.CompressionCodec> |
SHUFFLE_COMPRESSION_CODEC
The codec to be used when compressing shuffle data.
|
static ConfigOption<Boolean> |
TCP_CONNECTION_REUSE_ACROSS_JOBS_ENABLED
Whether to reuse tcp connections across multi jobs.
|
static ConfigOption<String> |
TRANSPORT_TYPE
Deprecated.
It is not recommended to use the option. It is deprecated in 1.20 and will be
totally removed in 2.0.
|
public static final ConfigOption<Integer> DATA_PORT
0
means that the TaskManager searches for a free port.public static final ConfigOption<String> DATA_BIND_PORT
public static final ConfigOption<Boolean> DATA_SSL_ENABLED
@Deprecated public static final ConfigOption<Boolean> BATCH_SHUFFLE_COMPRESSION_ENABLED
SHUFFLE_COMPRESSION_CODEC
to NONE to disable the
compression.Note: Data is compressed per buffer and compression can incur extra CPU overhead so it is more effective for IO bounded scenario when data compression ratio is high.
public static final ConfigOption<NettyShuffleEnvironmentOptions.CompressionCodec> SHUFFLE_COMPRESSION_CODEC
public static final ConfigOption<Boolean> NETWORK_DETAILED_METRICS
@Deprecated public static final ConfigOption<Integer> NETWORK_NUM_BUFFERS
TaskManagerOptions.NETWORK_MEMORY_FRACTION
, TaskManagerOptions.NETWORK_MEMORY_MIN
, and TaskManagerOptions.NETWORK_MEMORY_MAX
instead@Deprecated public static final ConfigOption<Float> NETWORK_BUFFERS_MEMORY_FRACTION
TaskManagerOptions.NETWORK_MEMORY_FRACTION
instead@Deprecated public static final ConfigOption<String> NETWORK_BUFFERS_MEMORY_MIN
TaskManagerOptions.NETWORK_MEMORY_MIN
instead@Deprecated public static final ConfigOption<String> NETWORK_BUFFERS_MEMORY_MAX
TaskManagerOptions.NETWORK_MEMORY_MAX
instead@Deprecated public static final ConfigOption<Integer> MAX_NUM_TCP_CONNECTIONS
@Experimental public static final ConfigOption<Integer> NETWORK_READ_MAX_REQUIRED_BUFFERS_PER_GATE
@Deprecated public static final ConfigOption<Integer> NETWORK_BUFFERS_PER_CHANNEL
@Deprecated public static final ConfigOption<Integer> NETWORK_EXTRA_BUFFERS_PER_GATE
public static final ConfigOption<Integer> NETWORK_SORT_SHUFFLE_MIN_BUFFERS
@Deprecated public static final ConfigOption<Integer> NETWORK_SORT_SHUFFLE_MIN_PARALLELISM
@Deprecated public static final ConfigOption<Integer> HYBRID_SHUFFLE_SPILLED_INDEX_REGION_GROUP_SIZE
@Deprecated public static final ConfigOption<Long> HYBRID_SHUFFLE_NUM_RETAINED_IN_MEMORY_REGIONS_MAX
@Deprecated public static final ConfigOption<Integer> NETWORK_MAX_BUFFERS_PER_CHANNEL
@Deprecated public static final ConfigOption<Integer> NETWORK_MAX_OVERDRAFT_BUFFERS_PER_GATE
@Deprecated public static final ConfigOption<Long> NETWORK_EXCLUSIVE_BUFFERS_REQUEST_TIMEOUT_MILLISECONDS
NETWORK_BUFFERS_REQUEST_TIMEOUT
instead.public static final ConfigOption<java.time.Duration> NETWORK_BUFFERS_REQUEST_TIMEOUT
@Deprecated @Experimental public static final ConfigOption<Boolean> NETWORK_HYBRID_SHUFFLE_ENABLE_NEW_MODE
@Experimental public static final ConfigOption<String> NETWORK_HYBRID_SHUFFLE_EXTERNAL_REMOTE_TIER_FACTORY_CLASS_NAME
@Experimental public static final ConfigOption<String> NETWORK_HYBRID_SHUFFLE_REMOTE_STORAGE_BASE_PATH
@Deprecated public static final ConfigOption<String> NETWORK_BLOCKING_SHUFFLE_TYPE
public static final ConfigOption<Boolean> TCP_CONNECTION_REUSE_ACROSS_JOBS_ENABLED
Note: To avoid connection leak, you must set MAX_NUM_TCP_CONNECTIONS
to a smaller
value before you enable tcp connection reuse.
@Deprecated public static final ConfigOption<Integer> NUM_ARENAS
@Deprecated public static final ConfigOption<Integer> NUM_THREADS_SERVER
@Deprecated public static final ConfigOption<Integer> NUM_THREADS_CLIENT
@Deprecated public static final ConfigOption<Integer> CONNECT_BACKLOG
public static final ConfigOption<Integer> CLIENT_CONNECT_TIMEOUT_SECONDS
public static final ConfigOption<Integer> NETWORK_RETRIES
@Deprecated public static final ConfigOption<Integer> SEND_RECEIVE_BUFFER_SIZE
@Deprecated public static final ConfigOption<String> TRANSPORT_TYPE
public static final ConfigOption<Integer> CLIENT_TCP_KEEP_IDLE_SECONDS
public static final ConfigOption<Integer> CLIENT_TCP_KEEP_INTERVAL_SECONDS
public static final ConfigOption<Integer> CLIENT_TCP_KEEP_COUNT
public static final ConfigOption<Integer> NETWORK_REQUEST_BACKOFF_INITIAL
public static final ConfigOption<Integer> NETWORK_REQUEST_BACKOFF_MAX
public static final ConfigOption<java.time.Duration> NETWORK_PARTITION_REQUEST_TIMEOUT
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.