org.apache.flink.runtime.taskmanager

Class ConsumableNotifyingResultPartitionWriterDecorator

java.lang.Object

org.apache.flink.runtime.taskmanager.ConsumableNotifyingResultPartitionWriterDecorator

All Implemented Interfaces:

AutoCloseable, AvailabilityProvider, ResultPartitionWriter

public class ConsumableNotifyingResultPartitionWriterDecorator
extends Object
implements ResultPartitionWriter

A wrapper of result partition writer for handling the logic of consumable notification.

Before a consuming task can request the result, it has to be deployed. The time of deployment depends on the PIPELINED vs. BLOCKING characteristic of the result partition. With pipelined results, receivers are deployed as soon as the first buffer is added to the result partition. With blocking results on the other hand, receivers are deployed after the partition is finished.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

AvailabilityProvider.AvailabilityHelper

Field Summary

Fields inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

AVAILABLE

Constructor Summary

Constructors

Constructor and Description
ConsumableNotifyingResultPartitionWriterDecorator(TaskActions taskActions, JobID jobId, ResultPartitionWriter partitionWriter, ResultPartitionConsumableNotifier partitionConsumableNotifier)

Method Summary

Modifier and Type	Method and Description
boolean	addBufferConsumer(BufferConsumer bufferConsumer, int subpartitionIndex, boolean isPriorityEvent) Adds the bufferConsumer to the subpartition with the given index.
void	close()
static ResultPartitionWriter[]	decorate(Collection<ResultPartitionDeploymentDescriptor> descs, ResultPartitionWriter[] partitionWriters, TaskActions taskActions, JobID jobId, ResultPartitionConsumableNotifier notifier)
void	fail(Throwable throwable) Fail the production of the partition.
void	finish() Successfully finish the production of the partition.
void	flush(int subpartitionIndex) Manually trigger consumption from enqueued BufferConsumers in one specified subpartition.
void	flushAll() Manually trigger consumption from enqueued BufferConsumers in all subpartitions.
CompletableFuture<?>	getAvailableFuture()
BufferBuilder	getBufferBuilder(int targetChannel) Requests a BufferBuilder from this partition for writing data.
int	getNumberOfSubpartitions()
int	getNumTargetKeyGroups()
ResultPartitionID	getPartitionId()
ResultSubpartition	getSubpartition(int subpartitionIndex) Returns the subpartition with the given index.
void	readRecoveredState(ChannelStateReader stateReader) Reads the previous output states with the given reader for unaligned checkpoint.
void	setup() Setup partition, potentially heavy-weight, blocking operation comparing to just creation.
BufferBuilder	tryGetBufferBuilder(int targetChannel) Try to request a BufferBuilder from this partition for writing data.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.io.network.api.writer.ResultPartitionWriter

addBufferConsumer

Methods inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

isApproximatelyAvailable, isAvailable

Constructor Detail

ConsumableNotifyingResultPartitionWriterDecorator

public ConsumableNotifyingResultPartitionWriterDecorator(TaskActions taskActions,
                                                         JobID jobId,
                                                         ResultPartitionWriter partitionWriter,
                                                         ResultPartitionConsumableNotifier partitionConsumableNotifier)

Method Detail

getBufferBuilder

public BufferBuilder getBufferBuilder(int targetChannel)
                               throws IOException,
                                      InterruptedException

Description copied from interface: ResultPartitionWriter

Requests a BufferBuilder from this partition for writing data.

Specified by:

getBufferBuilder in interface ResultPartitionWriter

Throws:

IOException

InterruptedException

tryGetBufferBuilder

public BufferBuilder tryGetBufferBuilder(int targetChannel)
                                  throws IOException

Description copied from interface: ResultPartitionWriter

Try to request a BufferBuilder from this partition for writing data.

Returns null if no buffer is available or the buffer provider has been destroyed.

Specified by:

tryGetBufferBuilder in interface ResultPartitionWriter

Throws:

IOException

getPartitionId

public ResultPartitionID getPartitionId()

Specified by:

getPartitionId in interface ResultPartitionWriter

getNumberOfSubpartitions

public int getNumberOfSubpartitions()

Specified by:

getNumberOfSubpartitions in interface ResultPartitionWriter

getNumTargetKeyGroups

public int getNumTargetKeyGroups()

Specified by:

getNumTargetKeyGroups in interface ResultPartitionWriter

setup

public void setup()
           throws IOException

Description copied from interface: ResultPartitionWriter

Setup partition, potentially heavy-weight, blocking operation comparing to just creation.

Specified by:

setup in interface ResultPartitionWriter

Throws:

IOException

getSubpartition

public ResultSubpartition getSubpartition(int subpartitionIndex)

Description copied from interface: ResultPartitionWriter

Returns the subpartition with the given index.

Specified by:

getSubpartition in interface ResultPartitionWriter

readRecoveredState

public void readRecoveredState(ChannelStateReader stateReader)
                        throws IOException,
                               InterruptedException

Description copied from interface: ResultPartitionWriter

Reads the previous output states with the given reader for unaligned checkpoint. It should be done before task processing the inputs.

Specified by:

readRecoveredState in interface ResultPartitionWriter

Throws:

IOException

InterruptedException

addBufferConsumer

public boolean addBufferConsumer(BufferConsumer bufferConsumer,
                                 int subpartitionIndex,
                                 boolean isPriorityEvent)
                          throws IOException

Description copied from interface: ResultPartitionWriter

Adds the bufferConsumer to the subpartition with the given index.

This method takes the ownership of the passed bufferConsumer and thus is responsible for releasing it's resources.

To avoid problems with data re-ordering, before adding new BufferConsumer the previously added one the given subpartitionIndex must be marked as BufferConsumer.isFinished().

Specified by:

addBufferConsumer in interface ResultPartitionWriter

Returns:

true if operation succeeded and bufferConsumer was enqueued for consumption.

Throws:

IOException

flushAll

public void flushAll()

Description copied from interface: ResultPartitionWriter

Manually trigger consumption from enqueued BufferConsumers in all subpartitions.

Specified by:

flushAll in interface ResultPartitionWriter

flush

public void flush(int subpartitionIndex)

Description copied from interface: ResultPartitionWriter

Manually trigger consumption from enqueued BufferConsumers in one specified subpartition.

Specified by:

flush in interface ResultPartitionWriter

finish

public void finish()
            throws IOException

Description copied from interface: ResultPartitionWriter

Successfully finish the production of the partition.

Closing of partition is still needed afterwards.

Specified by:

finish in interface ResultPartitionWriter

Throws:

IOException

fail

public void fail(Throwable throwable)

Description copied from interface: ResultPartitionWriter

Fail the production of the partition.

This method propagates non-null failure causes to consumers on a best-effort basis. This call also leads to the release of all resources associated with the partition. Closing of the partition is still needed afterwards if it has not been done before.

Specified by:

fail in interface ResultPartitionWriter

Parameters:

throwable - failure cause

getAvailableFuture

public CompletableFuture<?> getAvailableFuture()

Specified by:

getAvailableFuture in interface AvailabilityProvider

Returns:

a future that is completed if the respective provider is available.

close

public void close()
           throws Exception

Specified by:

close in interface AutoCloseable

Throws:

Exception

decorate

public static ResultPartitionWriter[] decorate(Collection<ResultPartitionDeploymentDescriptor> descs,
                                               ResultPartitionWriter[] partitionWriters,
                                               TaskActions taskActions,
                                               JobID jobId,
                                               ResultPartitionConsumableNotifier notifier)