org.apache.flink.runtime.shuffle

Class NettyShuffleMaster

java.lang.Object

org.apache.flink.runtime.shuffle.NettyShuffleMaster

All Implemented Interfaces:

AutoCloseable, ShuffleMaster<NettyShuffleDescriptor>

public class NettyShuffleMaster
extends Object
implements ShuffleMaster<NettyShuffleDescriptor>

Default ShuffleMaster for netty and local file based shuffle implementation.

Constructor Summary

Constructors

Constructor and Description
NettyShuffleMaster(Configuration conf)

Method Summary

Modifier and Type	Method and Description
MemorySize	computeShuffleMemorySizeForTask(TaskInputsOutputsDescriptor desc) JM announces network memory requirement from the calculating result of this method.
CompletableFuture<NettyShuffleDescriptor>	registerPartitionWithProducer(JobID jobID, PartitionDescriptor partitionDescriptor, ProducerDescriptor producerDescriptor) Asynchronously register a partition and its producer with the shuffle service.
void	releasePartitionExternally(ShuffleDescriptor shuffleDescriptor) Release any external resources occupied by the given partition.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.shuffle.ShuffleMaster

close, registerJob, start, unregisterJob

Constructor Detail

NettyShuffleMaster

public NettyShuffleMaster(Configuration conf)

Method Detail

registerPartitionWithProducer

public CompletableFuture<NettyShuffleDescriptor> registerPartitionWithProducer(JobID jobID,
                                                                               PartitionDescriptor partitionDescriptor,
                                                                               ProducerDescriptor producerDescriptor)

Description copied from interface: ShuffleMaster

Asynchronously register a partition and its producer with the shuffle service.

The returned shuffle descriptor is an internal handle which identifies the partition internally within the shuffle service. The descriptor should provide enough information to read from or write data to the partition.

Specified by:

registerPartitionWithProducer in interface ShuffleMaster<NettyShuffleDescriptor>

Parameters:

jobID - job ID of the corresponding job which registered the partition

partitionDescriptor - general job graph information about the partition

producerDescriptor - general producer information (location, execution id, connection info)

Returns:

future with the partition shuffle descriptor used for producer/consumer deployment and their data exchange.

releasePartitionExternally

public void releasePartitionExternally(ShuffleDescriptor shuffleDescriptor)

Description copied from interface: ShuffleMaster

Release any external resources occupied by the given partition.

This call triggers release of any resources which are occupied by the given partition in the external systems outside of the producer executor. This is mostly relevant for the batch jobs and blocking result partitions. The producer local resources are managed by ShuffleDescriptor.storesLocalResourcesOn() and ShuffleEnvironment.releasePartitionsLocally(Collection).

Specified by:

releasePartitionExternally in interface ShuffleMaster<NettyShuffleDescriptor>

Parameters:

shuffleDescriptor - shuffle descriptor of the result partition to release externally.

computeShuffleMemorySizeForTask

public MemorySize computeShuffleMemorySizeForTask(TaskInputsOutputsDescriptor desc)

JM announces network memory requirement from the calculating result of this method. Please note that the calculating algorithm depends on both I/O details of a vertex and network configuration, e.g. NettyShuffleEnvironmentOptions.NETWORK_BUFFERS_PER_CHANNEL and NettyShuffleEnvironmentOptions.NETWORK_EXTRA_BUFFERS_PER_GATE, which means we should always keep the consistency of configurations between JM, RM and TM in fine-grained resource management, thus to guarantee that the processes of memory announcing and allocating respect each other.

Specified by:

computeShuffleMemorySizeForTask in interface ShuffleMaster<NettyShuffleDescriptor>

Parameters:

desc - describes task inputs and outputs information for shuffle memory calculation.

Returns:

shuffle memory size for a task with the given TaskInputsOutputsDescriptor.