public class PipelinedSubpartition extends ResultSubpartition implements ChannelStateHolder
Whenever ResultSubpartition.add(BufferConsumer)
adds a finished BufferConsumer
or a second BufferConsumer
(in which case we will assume the first one finished), we will
notify
a read view created via ResultSubpartition.createReadView(BufferAvailabilityListener)
of new data availability. Except
by calling flush()
explicitly, we always only notify when the first finished buffer
turns up and then, the reader has to drain the buffers via pollBuffer()
until its return
value shows no more buffers being available. This results in a buffer queue which is either empty
or has an unfinished BufferConsumer
left from which the notifications will eventually
start again.
Explicit calls to flush()
will force this notification
for any BufferConsumer
present in the queue.
ResultSubpartition.BufferAndBacklog
ADD_BUFFER_ERROR_CODE, parent, subpartitionInfo
Modifier and Type | Method and Description |
---|---|
void |
abortCheckpoint(long checkpointId,
CheckpointException cause) |
void |
acknowledgeAllDataProcessed() |
int |
add(BufferConsumer bufferConsumer,
int partialRecordLength)
Adds the given buffer.
|
void |
alignedBarrierTimeout(long checkpointId) |
void |
bufferSize(int desirableNewBufferSize) |
PipelinedSubpartitionView |
createReadView(BufferAvailabilityListener availabilityListener) |
int |
finish()
Writing of data is finished.
|
void |
flush() |
ResultSubpartitionView.AvailabilityWithBacklog |
getAvailabilityAndBacklog(int numCreditsAvailable) |
int |
getBuffersInBacklogUnsafe()
Gets the number of non-event buffers in this subpartition.
|
long |
getChannelStateCheckpointId() |
int |
getNumberOfQueuedBuffers()
Get the current size of the queue.
|
protected long |
getTotalNumberOfBuffersUnsafe()
Gets the total numbers of buffers (data buffers plus events).
|
protected long |
getTotalNumberOfBytesUnsafe() |
boolean |
isReleased() |
boolean |
isSupportChannelStateRecover() |
void |
release() |
void |
setChannelStateWriter(ChannelStateWriter channelStateWriter)
Injects the
ChannelStateWriter . |
String |
toString() |
int |
unsynchronizedGetNumberOfQueuedBuffers()
Makes a best effort to get the current size of the queue.
|
add, getSubPartitionIndex, getSubpartitionInfo, onConsumedSubpartition
public void setChannelStateWriter(ChannelStateWriter channelStateWriter)
ChannelStateHolder
ChannelStateWriter
. Must only be called once.setChannelStateWriter
in interface ChannelStateHolder
public int add(BufferConsumer bufferConsumer, int partialRecordLength)
ResultSubpartition
The request may be executed synchronously, or asynchronously, depending on the implementation.
IMPORTANT: Before adding new BufferConsumer
previously added must
be in finished state. Because of the performance reasons, this is only enforced during the
data reading. Priority events can be added while the previous buffer consumer is still open,
in which case the open buffer consumer is overtaken.
add
in class ResultSubpartition
bufferConsumer
- the buffer to add (transferring ownership to this writer)partialRecordLength
- the length of bytes to skip in order to start with a complete
record, from position index 0 of the underlying .ResultSubpartition.ADD_BUFFER_ERROR_CODE
if
the add operation fails.public boolean isSupportChannelStateRecover()
public int finish() throws IOException
ResultSubpartition
finish
in class ResultSubpartition
IOException
public void alignedBarrierTimeout(long checkpointId) throws IOException
alignedBarrierTimeout
in class ResultSubpartition
IOException
public void abortCheckpoint(long checkpointId, CheckpointException cause)
abortCheckpoint
in class ResultSubpartition
public void release()
release
in class ResultSubpartition
public void acknowledgeAllDataProcessed()
public boolean isReleased()
isReleased
in class ResultSubpartition
public PipelinedSubpartitionView createReadView(BufferAvailabilityListener availabilityListener)
createReadView
in class ResultSubpartition
public ResultSubpartitionView.AvailabilityWithBacklog getAvailabilityAndBacklog(int numCreditsAvailable)
public int getNumberOfQueuedBuffers()
ResultSubpartition
getNumberOfQueuedBuffers
in class ResultSubpartition
public void bufferSize(int desirableNewBufferSize)
bufferSize
in class ResultSubpartition
public int unsynchronizedGetNumberOfQueuedBuffers()
ResultSubpartition
unsynchronizedGetNumberOfQueuedBuffers
in class ResultSubpartition
public void flush()
flush
in class ResultSubpartition
protected long getTotalNumberOfBuffersUnsafe()
ResultSubpartition
getTotalNumberOfBuffersUnsafe
in class ResultSubpartition
protected long getTotalNumberOfBytesUnsafe()
getTotalNumberOfBytesUnsafe
in class ResultSubpartition
public int getBuffersInBacklogUnsafe()
@VisibleForTesting public long getChannelStateCheckpointId()
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.