public class NettyShuffleEnvironmentConfiguration extends Object
Constructor and Description |
---|
NettyShuffleEnvironmentConfiguration(int numNetworkBuffers,
int networkBufferSize,
int partitionRequestInitialBackoff,
int partitionRequestMaxBackoff,
int networkBuffersPerChannel,
int floatingNetworkBuffersPerGate,
java.time.Duration requestSegmentsTimeout,
boolean isCreditBased,
boolean isNetworkDetailedMetrics,
NettyConfig nettyConfig,
String[] tempDirs,
BoundedBlockingSubpartitionType blockingSubpartitionType,
boolean forcePartitionReleaseOnConsumption) |
Modifier and Type | Method and Description |
---|---|
static long |
calculateNetworkBufferMemory(long totalJavaMemorySize,
Configuration config)
Calculates the amount of memory used for network buffers based on the total memory to use and
the according configuration parameters.
|
static long |
calculateNewNetworkBufferMemory(Configuration config,
long maxJvmHeapMemory)
Calculates the amount of memory used for network buffers inside the current JVM instance
based on the available heap or the max heap size and the according configuration parameters.
|
boolean |
equals(Object obj) |
int |
floatingNetworkBuffersPerGate() |
static NettyShuffleEnvironmentConfiguration |
fromConfiguration(Configuration configuration,
long maxJvmHeapMemory,
boolean localTaskManagerCommunication,
InetAddress taskManagerAddress)
Utility method to extract network related parameters from the configuration and to
sanity check them.
|
BoundedBlockingSubpartitionType |
getBlockingSubpartitionType() |
java.time.Duration |
getRequestSegmentsTimeout() |
String[] |
getTempDirs() |
int |
hashCode() |
static boolean |
hasNewNetworkConfig(Configuration config)
Returns whether the new network buffer memory configuration is present in the configuration
object, i.e.
|
boolean |
isCreditBased() |
boolean |
isForcePartitionReleaseOnConsumption() |
boolean |
isNetworkDetailedMetrics() |
NettyConfig |
nettyConfig() |
int |
networkBufferSize() |
int |
networkBuffersPerChannel() |
int |
numNetworkBuffers() |
int |
partitionRequestInitialBackoff() |
int |
partitionRequestMaxBackoff() |
String |
toString() |
public NettyShuffleEnvironmentConfiguration(int numNetworkBuffers, int networkBufferSize, int partitionRequestInitialBackoff, int partitionRequestMaxBackoff, int networkBuffersPerChannel, int floatingNetworkBuffersPerGate, java.time.Duration requestSegmentsTimeout, boolean isCreditBased, boolean isNetworkDetailedMetrics, @Nullable NettyConfig nettyConfig, String[] tempDirs, BoundedBlockingSubpartitionType blockingSubpartitionType, boolean forcePartitionReleaseOnConsumption)
public int numNetworkBuffers()
public int networkBufferSize()
public int partitionRequestInitialBackoff()
public int partitionRequestMaxBackoff()
public int networkBuffersPerChannel()
public int floatingNetworkBuffersPerGate()
public java.time.Duration getRequestSegmentsTimeout()
public NettyConfig nettyConfig()
public boolean isCreditBased()
public boolean isNetworkDetailedMetrics()
public String[] getTempDirs()
public BoundedBlockingSubpartitionType getBlockingSubpartitionType()
public boolean isForcePartitionReleaseOnConsumption()
public static NettyShuffleEnvironmentConfiguration fromConfiguration(Configuration configuration, long maxJvmHeapMemory, boolean localTaskManagerCommunication, InetAddress taskManagerAddress)
configuration
- configuration objectmaxJvmHeapMemory
- the maximum JVM heap size (in bytes)localTaskManagerCommunication
- true, to skip initializing the network stacktaskManagerAddress
- identifying the IP address under which the TaskManager will be accessible@VisibleForTesting public static long calculateNewNetworkBufferMemory(Configuration config, long maxJvmHeapMemory)
For containers or when started via scripts, if started with a memory limit and set to use off-heap memory, the maximum heap size for the JVM is adjusted accordingly and we are able to extract the intended values from this.
The following configuration parameters are involved:
TaskManagerOptions.MANAGED_MEMORY_SIZE
,TaskManagerOptions.MANAGED_MEMORY_FRACTION
,NettyShuffleEnvironmentOptions.NETWORK_BUFFERS_MEMORY_FRACTION
,NettyShuffleEnvironmentOptions.NETWORK_BUFFERS_MEMORY_MIN
,NettyShuffleEnvironmentOptions.NETWORK_BUFFERS_MEMORY_MAX
, andNettyShuffleEnvironmentOptions.NETWORK_NUM_BUFFERS
(fallback if the ones above do not exist)config
- configuration objectmaxJvmHeapMemory
- the maximum JVM heap size (in bytes)public static long calculateNetworkBufferMemory(long totalJavaMemorySize, Configuration config)
The following configuration parameters are involved:
.totalJavaMemorySize
- overall available memory to use (in bytes)config
- configuration object@VisibleForTesting public static boolean hasNewNetworkConfig(Configuration config)
config
- configuration objectCopyright © 2014–2020 The Apache Software Foundation. All rights reserved.