public class ResultPartition extends Object implements ResultPartitionWriter, BufferPoolOwner
This class is the runtime part of a logical IntermediateResultPartition
. Essentially,
a result partition is a collection of Buffer
instances. The buffers are organized in one
or more ResultSubpartition
instances, which further partition the data depending on the
number of consuming tasks and the data DistributionPattern
.
Tasks, which consume a result partition have to request one of its subpartitions. The request
happens either remotely (see RemoteInputChannel
) or locally (see LocalInputChannel
)
The life-cycle of each result partition has three (possibly overlapping) phases:
AvailabilityProvider.AvailabilityHelper
Modifier and Type | Field and Description |
---|---|
protected BufferCompressor |
bufferCompressor
Used to compress buffer to reduce IO.
|
protected static org.slf4j.Logger |
LOG |
int |
numTargetKeyGroups |
protected ResultPartitionID |
partitionId |
protected ResultPartitionManager |
partitionManager |
protected ResultPartitionType |
partitionType
Type of this partition.
|
protected ResultSubpartition[] |
subpartitions
The subpartitions of this partition.
|
AVAILABLE
Constructor and Description |
---|
ResultPartition(String owningTaskName,
int partitionIndex,
ResultPartitionID partitionId,
ResultPartitionType partitionType,
ResultSubpartition[] subpartitions,
int numTargetKeyGroups,
ResultPartitionManager partitionManager,
BufferCompressor bufferCompressor,
FunctionWithException<BufferPoolOwner,BufferPool,IOException> bufferPoolFactory) |
Modifier and Type | Method and Description |
---|---|
boolean |
addBufferConsumer(BufferConsumer bufferConsumer,
int subpartitionIndex,
boolean isPriorityEvent)
Adds the bufferConsumer to the subpartition with the given index.
|
void |
close() |
ResultSubpartitionView |
createSubpartitionView(int index,
BufferAvailabilityListener availabilityListener)
Returns the requested subpartition.
|
void |
fail(Throwable throwable)
Fail the production of the partition.
|
void |
finish()
Finishes the result partition.
|
void |
flush(int subpartitionIndex)
Manually trigger consumption from enqueued
BufferConsumers in one
specified subpartition. |
void |
flushAll()
Manually trigger consumption from enqueued
BufferConsumers in all
subpartitions. |
ResultSubpartition[] |
getAllPartitions() |
CompletableFuture<?> |
getAvailableFuture() |
BufferBuilder |
getBufferBuilder(int targetChannel)
Requests a
BufferBuilder from this partition for writing data. |
BufferPool |
getBufferPool() |
Throwable |
getFailureCause() |
int |
getNumberOfQueuedBuffers() |
int |
getNumberOfSubpartitions() |
int |
getNumTargetKeyGroups() |
String |
getOwningTaskName() |
ResultPartitionID |
getPartitionId() |
int |
getPartitionIndex() |
ResultPartitionType |
getPartitionType()
Returns the type of this result partition.
|
ResultSubpartition |
getSubpartition(int subpartitionIndex)
Returns the subpartition with the given index.
|
boolean |
isReleased()
Whether this partition is released.
|
void |
readRecoveredState(ChannelStateReader stateReader)
Reads the previous output states with the given reader for unaligned checkpoint.
|
void |
release() |
void |
release(Throwable cause)
Releases the result partition.
|
void |
releaseMemory(int toRelease)
Releases buffers held by this result partition.
|
void |
setup()
Registers a buffer pool with this result partition.
|
String |
toString() |
BufferBuilder |
tryGetBufferBuilder(int targetChannel)
Try to request a
BufferBuilder from this partition for writing data. |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
addBufferConsumer
isApproximatelyAvailable, isAvailable
protected static final org.slf4j.Logger LOG
protected final ResultPartitionID partitionId
protected final ResultPartitionType partitionType
protected final ResultSubpartition[] subpartitions
protected final ResultPartitionManager partitionManager
public final int numTargetKeyGroups
@Nullable protected final BufferCompressor bufferCompressor
public ResultPartition(String owningTaskName, int partitionIndex, ResultPartitionID partitionId, ResultPartitionType partitionType, ResultSubpartition[] subpartitions, int numTargetKeyGroups, ResultPartitionManager partitionManager, @Nullable BufferCompressor bufferCompressor, FunctionWithException<BufferPoolOwner,BufferPool,IOException> bufferPoolFactory)
public void setup() throws IOException
There is one pool for each result partition, which is shared by all its sub partitions.
The pool is registered with the partition *after* it as been constructed in order to
conform to the life-cycle of task registrations in the TaskExecutor
.
setup
in interface ResultPartitionWriter
IOException
public void readRecoveredState(ChannelStateReader stateReader) throws IOException, InterruptedException
ResultPartitionWriter
readRecoveredState
in interface ResultPartitionWriter
IOException
InterruptedException
public String getOwningTaskName()
public ResultPartitionID getPartitionId()
getPartitionId
in interface ResultPartitionWriter
public int getPartitionIndex()
public ResultSubpartition getSubpartition(int subpartitionIndex)
ResultPartitionWriter
getSubpartition
in interface ResultPartitionWriter
public int getNumberOfSubpartitions()
getNumberOfSubpartitions
in interface ResultPartitionWriter
public BufferPool getBufferPool()
public int getNumberOfQueuedBuffers()
public ResultPartitionType getPartitionType()
public BufferBuilder getBufferBuilder(int targetChannel) throws IOException, InterruptedException
ResultPartitionWriter
BufferBuilder
from this partition for writing data.getBufferBuilder
in interface ResultPartitionWriter
IOException
InterruptedException
public BufferBuilder tryGetBufferBuilder(int targetChannel) throws IOException
ResultPartitionWriter
BufferBuilder
from this partition for writing data.
Returns null
if no buffer is available or the buffer provider has been
destroyed.
tryGetBufferBuilder
in interface ResultPartitionWriter
IOException
public boolean addBufferConsumer(BufferConsumer bufferConsumer, int subpartitionIndex, boolean isPriorityEvent) throws IOException
ResultPartitionWriter
This method takes the ownership of the passed bufferConsumer
and thus is
responsible for releasing it's resources.
To avoid problems with data re-ordering, before adding new BufferConsumer
the
previously added one the given subpartitionIndex
must be marked as BufferConsumer.isFinished()
.
addBufferConsumer
in interface ResultPartitionWriter
IOException
public void flushAll()
ResultPartitionWriter
BufferConsumers
in all
subpartitions.flushAll
in interface ResultPartitionWriter
public void flush(int subpartitionIndex)
ResultPartitionWriter
BufferConsumers
in one
specified subpartition.flush
in interface ResultPartitionWriter
public void finish() throws IOException
After this operation, it is not possible to add further data to the result partition.
For BLOCKING results, this will trigger the deployment of consuming tasks.
finish
in interface ResultPartitionWriter
IOException
public void release()
public void release(Throwable cause)
public void close()
close
in interface AutoCloseable
public void fail(@Nullable Throwable throwable)
ResultPartitionWriter
This method propagates non-null
failure causes to consumers on a best-effort
basis. This call also leads to the release of all resources associated with the partition.
Closing of the partition is still needed afterwards if it has not been done before.
fail
in interface ResultPartitionWriter
throwable
- failure causepublic ResultSubpartitionView createSubpartitionView(int index, BufferAvailabilityListener availabilityListener) throws IOException
IOException
public Throwable getFailureCause()
public int getNumTargetKeyGroups()
getNumTargetKeyGroups
in interface ResultPartitionWriter
public void releaseMemory(int toRelease) throws IOException
This is a callback from the buffer pool, which is registered for result partitions, which are back pressure-free.
releaseMemory
in interface BufferPoolOwner
IOException
public boolean isReleased()
A partition is released when each subpartition is either consumed and communication is closed by consumer or failed. A partition is also released if task is cancelled.
public CompletableFuture<?> getAvailableFuture()
getAvailableFuture
in interface AvailabilityProvider
public ResultSubpartition[] getAllPartitions()
Copyright © 2014–2021 The Apache Software Foundation. All rights reserved.