org.apache.flink.runtime.shuffle

Class NettyShuffleUtils

java.lang.Object

org.apache.flink.runtime.shuffle.NettyShuffleUtils

public class NettyShuffleUtils
extends Object

Utils to calculate network memory requirement of a vertex from network configuration and details of input and output. The methods help to decide the volume of buffer pools when initializing shuffle environment and also guide network memory announcing in fine-grained resource management.

Method Summary

Modifier and Type	Method and Description
static int	computeNetworkBuffersForAnnouncing(int numBuffersPerChannel, int numFloatingBuffersPerGate, Optional<Integer> maxRequiredBuffersPerGate, int sortShuffleMinParallelism, int sortShuffleMinBuffers, Map<IntermediateDataSetID,Integer> inputChannelNums, Map<IntermediateDataSetID,Integer> partitionReuseCount, Map<IntermediateDataSetID,Integer> subpartitionNums, Map<IntermediateDataSetID,ResultPartitionType> inputPartitionTypes, Map<IntermediateDataSetID,ResultPartitionType> partitionTypes)
static org.apache.commons.lang3.tuple.Pair<Integer,Integer>	getMinMaxFloatingBuffersPerInputGate(int numFloatingBuffersPerGate) Calculates and returns the floating network buffer pool size used by the input gate.
static org.apache.commons.lang3.tuple.Pair<Integer,Integer>	getMinMaxNetworkBuffersPerResultPartition(int configuredNetworkBuffersPerChannel, int numFloatingBuffersPerGate, int sortShuffleMinParallelism, int sortShuffleMinBuffers, int numSubpartitions, boolean enableTieredStorage, int tieredStoreExclusiveBuffers, ResultPartitionType type) Calculates and returns local network buffer pool size used by the result partition.
static int	getNetworkBuffersPerInputChannel(int configuredNetworkBuffersPerChannel) Calculates and returns the number of required exclusive network buffers per input channel.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

getNetworkBuffersPerInputChannel

public static int getNetworkBuffersPerInputChannel(int configuredNetworkBuffersPerChannel)

Calculates and returns the number of required exclusive network buffers per input channel.

getMinMaxFloatingBuffersPerInputGate

public static org.apache.commons.lang3.tuple.Pair<Integer,Integer> getMinMaxFloatingBuffersPerInputGate(int numFloatingBuffersPerGate)

Calculates and returns the floating network buffer pool size used by the input gate. The left/right value of the returned pair represent the min/max buffers require by the pool.

getMinMaxNetworkBuffersPerResultPartition

public static org.apache.commons.lang3.tuple.Pair<Integer,Integer> getMinMaxNetworkBuffersPerResultPartition(int configuredNetworkBuffersPerChannel,
                                                                                                             int numFloatingBuffersPerGate,
                                                                                                             int sortShuffleMinParallelism,
                                                                                                             int sortShuffleMinBuffers,
                                                                                                             int numSubpartitions,
                                                                                                             boolean enableTieredStorage,
                                                                                                             int tieredStoreExclusiveBuffers,
                                                                                                             ResultPartitionType type)

Calculates and returns local network buffer pool size used by the result partition. The left/right value of the returned pair represent the min/max buffers require by the pool.

computeNetworkBuffersForAnnouncing

public static int computeNetworkBuffersForAnnouncing(int numBuffersPerChannel,
                                                     int numFloatingBuffersPerGate,
                                                     Optional<Integer> maxRequiredBuffersPerGate,
                                                     int sortShuffleMinParallelism,
                                                     int sortShuffleMinBuffers,
                                                     Map<IntermediateDataSetID,Integer> inputChannelNums,
                                                     Map<IntermediateDataSetID,Integer> partitionReuseCount,
                                                     Map<IntermediateDataSetID,Integer> subpartitionNums,
                                                     Map<IntermediateDataSetID,ResultPartitionType> inputPartitionTypes,
                                                     Map<IntermediateDataSetID,ResultPartitionType> partitionTypes)