public class PipelinedSubpartition extends ResultSubpartition
Whenever ResultSubpartition.add(BufferConsumer, boolean)
adds a finished BufferConsumer
or a second BufferConsumer
(in which case we will assume the first one
finished), we will notify
a read view
created via ResultSubpartition.createReadView(BufferAvailabilityListener)
of new data
availability. Except by calling flush()
explicitly, we always only notify when the first
finished buffer turns up and then, the reader has to drain the buffers via pollBuffer()
until its return value shows no more buffers being available. This results in a buffer queue
which is either empty or has an unfinished BufferConsumer
left from which the
notifications will eventually start again.
Explicit calls to flush()
will force this notification
for any BufferConsumer
present in the queue.
ResultSubpartition.BufferAndBacklog
parent, subpartitionInfo
Modifier and Type | Method and Description |
---|---|
boolean |
add(BufferConsumer bufferConsumer,
boolean isPriorityEvent)
Adds the given buffer.
|
PipelinedSubpartitionView |
createReadView(BufferAvailabilityListener availabilityListener) |
void |
finish() |
void |
flush() |
int |
getBuffersInBacklog()
Gets the number of non-event buffers in this subpartition.
|
protected long |
getTotalNumberOfBuffers()
Gets the total numbers of buffers (data buffers plus events).
|
protected long |
getTotalNumberOfBytes() |
boolean |
isAvailable(int numCreditsAvailable) |
boolean |
isReleased() |
void |
readRecoveredState(ChannelStateReader stateReader) |
void |
release() |
int |
releaseMemory() |
List<Buffer> |
requestInflightBufferSnapshot() |
String |
toString() |
int |
unsynchronizedGetNumberOfQueuedBuffers()
Makes a best effort to get the current size of the queue.
|
add, canBeCompressed, getSubPartitionIndex, getSubpartitionInfo, onConsumedSubpartition
public void readRecoveredState(ChannelStateReader stateReader) throws IOException, InterruptedException
readRecoveredState
in class ResultSubpartition
IOException
InterruptedException
public boolean add(BufferConsumer bufferConsumer, boolean isPriorityEvent) throws IOException
ResultSubpartition
The request may be executed synchronously, or asynchronously, depending on the implementation.
IMPORTANT: Before adding new BufferConsumer
previously added must
be in finished state. Because of the performance reasons, this is only enforced during the
data reading.
add
in class ResultSubpartition
bufferConsumer
- the buffer to add (transferring ownership to this writer)IOException
- thrown in case of errors while adding the bufferpublic void finish() throws IOException
finish
in class ResultSubpartition
IOException
public List<Buffer> requestInflightBufferSnapshot()
requestInflightBufferSnapshot
in class ResultSubpartition
public void release()
release
in class ResultSubpartition
public int releaseMemory()
public boolean isReleased()
isReleased
in class ResultSubpartition
public PipelinedSubpartitionView createReadView(BufferAvailabilityListener availabilityListener) throws IOException
createReadView
in class ResultSubpartition
IOException
public boolean isAvailable(int numCreditsAvailable)
public int unsynchronizedGetNumberOfQueuedBuffers()
ResultSubpartition
unsynchronizedGetNumberOfQueuedBuffers
in class ResultSubpartition
public void flush()
flush
in class ResultSubpartition
protected long getTotalNumberOfBuffers()
ResultSubpartition
getTotalNumberOfBuffers
in class ResultSubpartition
protected long getTotalNumberOfBytes()
getTotalNumberOfBytes
in class ResultSubpartition
@VisibleForTesting public int getBuffersInBacklog()
Beware: This method should only be used in tests in non-concurrent access scenarios since it does not make any concurrency guarantees.
Copyright © 2014–2021 The Apache Software Foundation. All rights reserved.