org.apache.flink.runtime.io.network.partition

Interface ResultSubpartitionView

All Known Implementing Classes:

BoundedBlockingSubpartitionDirectTransferReader, HsSubpartitionConsumer, NoOpResultSubpartitionView, PipelinedApproximateSubpartitionView, PipelinedSubpartitionView, TieredStorageResultSubpartitionView, UnionResultSubpartitionView

public interface ResultSubpartitionView

A view to consume a ResultSubpartition instance.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	ResultSubpartitionView.AvailabilityWithBacklog Availability of the ResultSubpartitionView and the backlog in the corresponding ResultSubpartition.

Method Summary

Modifier and Type	Method and Description
void	acknowledgeAllDataProcessed()
ResultSubpartitionView.AvailabilityWithBacklog	getAvailabilityAndBacklog(boolean isCreditAvailable) Get the availability and backlog of the view.
Throwable	getFailureCause() ResultSubpartitionView can decide whether the failure cause should be reported to consumer as failure (primary failure) or ProducerFailedException (secondary failure).
ResultSubpartition.BufferAndBacklog	getNextBuffer() Returns the next Buffer instance of this queue iterator.
int	getNumberOfQueuedBuffers()
boolean	isReleased()
void	notifyDataAvailable()
void	notifyNewBufferSize(int newBufferSize)
default void	notifyPriorityEvent(int priorityBufferNumber)
default void	notifyRequiredSegmentId(int subpartitionId, int segmentId) In tiered storage shuffle mode, only required segments will be sent to prevent the redundant buffer usage.
int	peekNextBufferSubpartitionId() Returns the index of the subpartition where the next buffer locates, or -1 if there is no buffer available and the subpartition to be consumed is not determined.
void	releaseAllResources()
void	resumeConsumption()
int	unsynchronizedGetNumberOfQueuedBuffers()

Method Detail

getNextBuffer

@Nullable
ResultSubpartition.BufferAndBacklog getNextBuffer()
                                                     throws IOException

Returns the next Buffer instance of this queue iterator.

If there is currently no instance available, it will return null. This might happen for example when a pipelined queue producer is slower than the consumer or a spilled queue needs to read in more data.

Important: The consumer has to make sure that each buffer instance will eventually be recycled with Buffer.recycleBuffer() after it has been consumed.

Throws:

IOException

notifyDataAvailable

void notifyDataAvailable()

notifyPriorityEvent

default void notifyPriorityEvent(int priorityBufferNumber)

releaseAllResources

void releaseAllResources()
                  throws IOException

Throws:

IOException

isReleased

boolean isReleased()

resumeConsumption

void resumeConsumption()

acknowledgeAllDataProcessed

void acknowledgeAllDataProcessed()

getFailureCause

Throwable getFailureCause()

ResultSubpartitionView can decide whether the failure cause should be reported to consumer as failure (primary failure) or ProducerFailedException (secondary failure). Secondary failure can be reported only if producer (upstream task) is guaranteed to failover.

BEWARE: Incorrectly reporting failure cause as primary failure, can hide the root cause of the failure from the user.

getAvailabilityAndBacklog

ResultSubpartitionView.AvailabilityWithBacklog getAvailabilityAndBacklog(boolean isCreditAvailable)

Get the availability and backlog of the view. The availability represents if the view is ready to get buffer from it. The backlog represents the number of available data buffers.

Parameters:

isCreditAvailable - the availability of credits for this ResultSubpartitionView.

Returns:

availability and backlog.

unsynchronizedGetNumberOfQueuedBuffers

int unsynchronizedGetNumberOfQueuedBuffers()

getNumberOfQueuedBuffers

int getNumberOfQueuedBuffers()

notifyNewBufferSize

void notifyNewBufferSize(int newBufferSize)

notifyRequiredSegmentId

default void notifyRequiredSegmentId(int subpartitionId,
                                     int segmentId)

In tiered storage shuffle mode, only required segments will be sent to prevent the redundant buffer usage. Downstream will notify the upstream by this method to send required segments.

Parameters:

subpartitionId - The id of the corresponding subpartition.

segmentId - The id of required segment.

peekNextBufferSubpartitionId

int peekNextBufferSubpartitionId()
                          throws IOException

Returns the index of the subpartition where the next buffer locates, or -1 if there is no buffer available and the subpartition to be consumed is not determined.

Throws:

IOException