public abstract class BufferWritingResultPartition extends ResultPartition
ResultPartition
which writes buffers directly to ResultSubpartition
s. This is
in contrast to implementations where records are written to a joint structure, from which the
subpartitions draw the data after the write phase is finished, for example the sort-based
partitioning.
To avoid confusion: On the read side, all subpartitions return buffers (and backlog) to be transported through the network.
AvailabilityProvider.AvailabilityHelper
Modifier and Type | Field and Description |
---|---|
protected ResultSubpartition[] |
subpartitions
The subpartitions of this partition.
|
bufferCompressor, bufferPool, LOG, numBuffersOut, numBytesOut, numSubpartitions, partitionId, partitionManager, partitionType, resultPartitionBytes
AVAILABLE
Constructor and Description |
---|
BufferWritingResultPartition(String owningTaskName,
int partitionIndex,
ResultPartitionID partitionId,
ResultPartitionType partitionType,
ResultSubpartition[] subpartitions,
int numTargetKeyGroups,
ResultPartitionManager partitionManager,
BufferCompressor bufferCompressor,
SupplierWithException<BufferPool,IOException> bufferPoolFactory) |
Modifier and Type | Method and Description |
---|---|
void |
abortCheckpoint(long checkpointId,
CheckpointException cause)
Abort the checkpoint.
|
protected int |
addToSubpartition(int targetSubpartition,
BufferConsumer bufferConsumer,
int partialRecordLength) |
void |
alignedBarrierTimeout(long checkpointId)
Timeout the aligned barrier to unaligned barrier.
|
void |
broadcastEvent(AbstractEvent event,
boolean isPriorityEvent)
Writes the given
AbstractEvent to all subpartitions. |
void |
broadcastRecord(ByteBuffer record)
Writes the given serialized record to all subpartitions.
|
void |
close()
Closes the partition writer which releases the allocated resource, for example the buffer
pool.
|
protected ResultSubpartitionView |
createSubpartitionView(int subpartitionIndex,
BufferAvailabilityListener availabilityListener)
Returns a reader for the subpartition with the given index.
|
void |
emitRecord(ByteBuffer record,
int targetSubpartition)
Writes the given serialized record to the target subpartition.
|
void |
finish()
Finishes the result partition.
|
protected void |
flushAllSubpartitions(boolean finishProducers) |
protected void |
flushSubpartition(int targetSubpartition,
boolean finishProducers) |
ResultSubpartition[] |
getAllPartitions() |
TimerGauge |
getHardBackPressuredTimeMsPerSecond() |
int |
getNumberOfQueuedBuffers()
Returns the total number of queued buffers of all subpartitions.
|
int |
getNumberOfQueuedBuffers(int targetSubpartition)
Returns the number of queued buffers of the given target subpartition.
|
long |
getSizeOfQueuedBuffersUnsafe()
Returns the total size in bytes of queued buffers of all subpartitions.
|
protected void |
releaseInternal()
Releases all produced data including both those stored in memory and persisted on disk.
|
void |
setMetricGroup(TaskIOMetricGroup metrics)
Sets the metric group for the
ResultPartitionWriter . |
protected void |
setupInternal()
Do the subclass's own setup operation.
|
canBeCompressed, checkInProduceState, createSubpartitionView, fail, getAllDataProcessedFuture, getAvailableFuture, getBufferPool, getFailureCause, getNumberOfSubpartitions, getNumTargetKeyGroups, getOwningTaskName, getPartitionId, getPartitionIndex, getPartitionManager, getPartitionType, getResultPartitionBytes, isFinished, isNumberOfPartitionConsumerUndefined, isNumberOfPartitionConsumerUndefined, isReleased, notifyEndOfData, onSubpartitionAllDataProcessed, release, release, setMaxOverdraftBuffersPerGate, setup, toString
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
flush, flushAll
and, isApproximatelyAvailable, isAvailable, or
protected final ResultSubpartition[] subpartitions
public BufferWritingResultPartition(String owningTaskName, int partitionIndex, ResultPartitionID partitionId, ResultPartitionType partitionType, ResultSubpartition[] subpartitions, int numTargetKeyGroups, ResultPartitionManager partitionManager, @Nullable BufferCompressor bufferCompressor, SupplierWithException<BufferPool,IOException> bufferPoolFactory)
protected void setupInternal() throws IOException
ResultPartition
setupInternal
in class ResultPartition
IOException
public int getNumberOfQueuedBuffers()
ResultPartition
getNumberOfQueuedBuffers
in class ResultPartition
public long getSizeOfQueuedBuffersUnsafe()
ResultPartition
getSizeOfQueuedBuffersUnsafe
in class ResultPartition
public int getNumberOfQueuedBuffers(int targetSubpartition)
ResultPartition
getNumberOfQueuedBuffers
in class ResultPartition
protected void flushSubpartition(int targetSubpartition, boolean finishProducers)
protected void flushAllSubpartitions(boolean finishProducers)
public void emitRecord(ByteBuffer record, int targetSubpartition) throws IOException
ResultPartitionWriter
IOException
public void broadcastRecord(ByteBuffer record) throws IOException
ResultPartitionWriter
IOException
public void broadcastEvent(AbstractEvent event, boolean isPriorityEvent) throws IOException
ResultPartitionWriter
AbstractEvent
to all subpartitions.IOException
public void alignedBarrierTimeout(long checkpointId) throws IOException
ResultPartitionWriter
IOException
public void abortCheckpoint(long checkpointId, CheckpointException cause)
ResultPartitionWriter
public void setMetricGroup(TaskIOMetricGroup metrics)
ResultPartitionWriter
ResultPartitionWriter
.setMetricGroup
in interface ResultPartitionWriter
setMetricGroup
in class ResultPartition
protected ResultSubpartitionView createSubpartitionView(int subpartitionIndex, BufferAvailabilityListener availabilityListener) throws IOException
ResultPartition
Given that the function to merge outputs from multiple subpartition views is supported
uniformly in UnionResultSubpartitionView
, subclasses of ResultPartition
only
needs to take care of creating subpartition view for a single subpartition.
createSubpartitionView
in class ResultPartition
IOException
public void finish() throws IOException
ResultPartition
After this operation, it is not possible to add further data to the result partition.
For BLOCKING results, this will trigger the deployment of consuming tasks.
finish
in interface ResultPartitionWriter
finish
in class ResultPartition
IOException
protected void releaseInternal()
ResultPartition
releaseInternal
in class ResultPartition
public void close()
ResultPartitionWriter
close
in interface AutoCloseable
close
in interface ResultPartitionWriter
close
in class ResultPartition
protected int addToSubpartition(int targetSubpartition, BufferConsumer bufferConsumer, int partialRecordLength) throws IOException
IOException
@VisibleForTesting public TimerGauge getHardBackPressuredTimeMsPerSecond()
@VisibleForTesting public ResultSubpartition[] getAllPartitions()
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.