org.apache.flink.runtime.shuffle

Interface ShuffleMaster<T extends ShuffleDescriptor>

Type Parameters:

T - partition shuffle descriptor used for producer/consumer deployment and their data exchange.

All Known Implementing Classes:

NettyShuffleMaster

public interface ShuffleMaster<T extends ShuffleDescriptor>

Intermediate result partition registry to use in JobMaster.

Method Summary

Modifier and Type	Method and Description
CompletableFuture<T>	registerPartitionWithProducer(PartitionDescriptor partitionDescriptor, ProducerDescriptor producerDescriptor) Asynchronously register a partition and its producer with the shuffle service.
void	releasePartitionExternally(ShuffleDescriptor shuffleDescriptor) Release any external resources occupied by the given partition.

Method Detail

registerPartitionWithProducer

CompletableFuture<T> registerPartitionWithProducer(PartitionDescriptor partitionDescriptor,
                                                   ProducerDescriptor producerDescriptor)

Asynchronously register a partition and its producer with the shuffle service.

IMPORTANT: the returned future must be completed due to limitations in the default scheduler.

The returned shuffle descriptor is an internal handle which identifies the partition internally within the shuffle service. The descriptor should provide enough information to read from or write data to the partition.

Parameters:

partitionDescriptor - general job graph information about the partition

producerDescriptor - general producer information (location, execution id, connection info)

Returns:

future with the partition shuffle descriptor used for producer/consumer deployment and their data exchange.

releasePartitionExternally

void releasePartitionExternally(ShuffleDescriptor shuffleDescriptor)

Release any external resources occupied by the given partition.

This call triggers release of any resources which are occupied by the given partition in the external systems outside of the producer executor. This is mostly relevant for the batch jobs and blocking result partitions. The producer local resources are managed by ShuffleDescriptor.storesLocalResourcesOn() and ShuffleEnvironment.releasePartitionsLocally(Collection).

Parameters:

shuffleDescriptor - shuffle descriptor of the result partition to release externally.