org.apache.flink.runtime.io.network.partition

Class ResultPartition

java.lang.Object

org.apache.flink.runtime.io.network.partition.ResultPartition

All Implemented Interfaces:

ResultPartitionWriter, BufferPoolOwner

public class ResultPartition
extends Object
implements ResultPartitionWriter, BufferPoolOwner

A result partition for data produced by a single task.

This class is the runtime part of a logical IntermediateResultPartition. Essentially, a result partition is a collection of Buffer instances. The buffers are organized in one or more ResultSubpartition instances, which further partition the data depending on the number of consuming tasks and the data DistributionPattern.

Tasks, which consume a result partition have to request one of its subpartitions. The request happens either remotely (see RemoteInputChannel) or locally (see LocalInputChannel)

Life-cycle

The life-cycle of each result partition has three (possibly overlapping) phases:

1. Produce:
2. Consume:
3. Release:

Lazy deployment and updates of consuming tasks

Before a consuming task can request the result, it has to be deployed. The time of deployment depends on the PIPELINED vs. BLOCKING characteristic of the result partition. With pipelined results, receivers are deployed as soon as the first buffer is added to the result partition. With blocking results on the other hand, receivers are deployed after the partition is finished.

Buffer management

State management

Field Summary

Fields

Modifier and Type	Field and Description
int	numTargetKeyGroups

Constructor Summary

Constructors

Constructor and Description
ResultPartition(String owningTaskName, TaskActions taskActions, JobID jobId, ResultPartitionID partitionId, ResultPartitionType partitionType, int numberOfSubpartitions, int numTargetKeyGroups, ResultPartitionManager partitionManager, ResultPartitionConsumableNotifier partitionConsumableNotifier, IOManager ioManager, boolean sendScheduleOrUpdateConsumersMessage)

Method Summary

Modifier and Type	Method and Description
void	addBufferConsumer(BufferConsumer bufferConsumer, int subpartitionIndex) Adds the bufferConsumer to the subpartition with the given index.
ResultSubpartitionView	createSubpartitionView(int index, BufferAvailabilityListener availabilityListener) Returns the requested subpartition.
void	destroyBufferPool()
void	finish() Finishes the result partition.
void	flush(int subpartitionIndex) Manually trigger consumption from enqueued BufferConsumers in one specified subpartition.
void	flushAll() Manually trigger consumption from enqueued BufferConsumers in all subpartitions.
BufferPool	getBufferPool()
BufferProvider	getBufferProvider()
Throwable	getFailureCause()
JobID	getJobId()
int	getNumberOfQueuedBuffers()
int	getNumberOfSubpartitions()
int	getNumTargetKeyGroups()
String	getOwningTaskName()
ResultPartitionID	getPartitionId()
ResultPartitionType	getPartitionType() Returns the type of this result partition.
boolean	isReleased() Whether this partition is released.
void	registerBufferPool(BufferPool bufferPool) Registers a buffer pool with this result partition.
void	release()
void	release(Throwable cause) Releases the result partition.
void	releaseMemory(int toRelease) Releases buffers held by this result partition.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

numTargetKeyGroups

public final int numTargetKeyGroups

Constructor Detail

ResultPartition

public ResultPartition(String owningTaskName,
                       TaskActions taskActions,
                       JobID jobId,
                       ResultPartitionID partitionId,
                       ResultPartitionType partitionType,
                       int numberOfSubpartitions,
                       int numTargetKeyGroups,
                       ResultPartitionManager partitionManager,
                       ResultPartitionConsumableNotifier partitionConsumableNotifier,
                       IOManager ioManager,
                       boolean sendScheduleOrUpdateConsumersMessage)

Method Detail

registerBufferPool

public void registerBufferPool(BufferPool bufferPool)

Registers a buffer pool with this result partition.

There is one pool for each result partition, which is shared by all its sub partitions.

The pool is registered with the partition *after* it as been constructed in order to conform to the life-cycle of task registrations in the TaskManager.

getJobId

public JobID getJobId()

getOwningTaskName

public String getOwningTaskName()

getPartitionId

public ResultPartitionID getPartitionId()

Specified by:

getPartitionId in interface ResultPartitionWriter

getNumberOfSubpartitions

public int getNumberOfSubpartitions()

Specified by:

getNumberOfSubpartitions in interface ResultPartitionWriter

getBufferProvider

public BufferProvider getBufferProvider()

Specified by:

getBufferProvider in interface ResultPartitionWriter

getBufferPool

public BufferPool getBufferPool()

getNumberOfQueuedBuffers

public int getNumberOfQueuedBuffers()

getPartitionType

public ResultPartitionType getPartitionType()

Returns the type of this result partition.

Returns:

result partition type

addBufferConsumer

public void addBufferConsumer(BufferConsumer bufferConsumer,
                              int subpartitionIndex)
                       throws IOException

Description copied from interface: ResultPartitionWriter

Adds the bufferConsumer to the subpartition with the given index.

For PIPELINED ResultPartitionTypes, this will trigger the deployment of consuming tasks after the first buffer has been added.

This method takes the ownership of the passed bufferConsumer and thus is responsible for releasing it's resources.

To avoid problems with data re-ordering, before adding new BufferConsumer the previously added one the given subpartitionIndex must be marked as BufferConsumer.isFinished().

Specified by:

addBufferConsumer in interface ResultPartitionWriter

Throws:

IOException

flushAll

public void flushAll()

Description copied from interface: ResultPartitionWriter

Manually trigger consumption from enqueued BufferConsumers in all subpartitions.

Specified by:

flushAll in interface ResultPartitionWriter

flush

public void flush(int subpartitionIndex)

Description copied from interface: ResultPartitionWriter

Manually trigger consumption from enqueued BufferConsumers in one specified subpartition.

Specified by:

flush in interface ResultPartitionWriter

finish

public void finish()
            throws IOException

Finishes the result partition.

After this operation, it is not possible to add further data to the result partition.

For BLOCKING results, this will trigger the deployment of consuming tasks.

Throws:

IOException

release

public void release()

release

public void release(Throwable cause)

Releases the result partition.

destroyBufferPool

public void destroyBufferPool()

createSubpartitionView

public ResultSubpartitionView createSubpartitionView(int index,
                                                     BufferAvailabilityListener availabilityListener)
                                              throws IOException

Returns the requested subpartition.

Throws:

IOException

getFailureCause

public Throwable getFailureCause()

getNumTargetKeyGroups

public int getNumTargetKeyGroups()

Specified by:

getNumTargetKeyGroups in interface ResultPartitionWriter

releaseMemory

public void releaseMemory(int toRelease)
                   throws IOException

Releases buffers held by this result partition.

This is a callback from the buffer pool, which is registered for result partitions, which are back pressure-free.

Specified by:

releaseMemory in interface BufferPoolOwner

Throws:

IOException

isReleased

public boolean isReleased()

Whether this partition is released.

A partition is released when each subpartition is either consumed and communication is closed by consumer or failed. A partition is also released if task is cancelled.

toString

public String toString()

Overrides:

toString in class Object