E
- The type of events returned by the reader.IterT
- The type of the iterator that produces the events. This type exists to make the
conversion between iterator and IteratorSourceSplit
type safe.SplitT
- The concrete type of the IteratorSourceSplit
that creates and converts
the iterator that produces this reader's elements.@Public public abstract class IteratorSourceReaderBase<E,O,IterT extends Iterator<E>,SplitT extends IteratorSourceSplit<E,IterT>> extends Object implements SourceReader<O,SplitT>
SourceReader
that returns the values of an iterator, supplied via an IteratorSourceSplit
.
The IteratorSourceSplit
is also responsible for taking the current iterator and
turning it back into a split for checkpointing.
Modifier and Type | Field and Description |
---|---|
protected CompletableFuture<Void> |
availability
The availability future.
|
protected SplitT |
currentSplit
The split whose data we return.
|
protected IterT |
iterator
The iterator producing data.
|
Constructor and Description |
---|
IteratorSourceReaderBase(SourceReaderContext context) |
Modifier and Type | Method and Description |
---|---|
void |
addSplits(List<SplitT> splits)
Adds a list of splits for this reader to read.
|
void |
close() |
protected abstract O |
convert(E value) |
protected void |
finishSplit() |
CompletableFuture<Void> |
isAvailable()
Returns a future that signals that data is available from the reader.
|
void |
notifyNoMoreSplits()
This method is called when the reader is notified that it will not receive any further
splits.
|
InputStatus |
pollNext(ReaderOutput<O> output)
Poll the next available record into the
ReaderOutput . |
List<SplitT> |
snapshotState(long checkpointId)
Checkpoint on the state of the source.
|
void |
start()
Start the reader.
|
protected void |
start(SourceReaderContext context) |
protected InputStatus |
tryMoveToNextSplit() |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
handleSourceEvents, notifyCheckpointComplete, pauseOrResumeSplits
notifyCheckpointAborted
protected CompletableFuture<Void> availability
@Nullable protected IterT extends Iterator<E> iterator
currentSplit
field.public IteratorSourceReaderBase(SourceReaderContext context)
public void start()
SourceReader
start
in interface SourceReader<O,SplitT extends IteratorSourceSplit<E,IterT>>
protected void start(SourceReaderContext context)
public InputStatus pollNext(ReaderOutput<O> output)
SourceReader
ReaderOutput
.
The implementation must make sure this method is non-blocking.
Although the implementation can emit multiple records into the given ReaderOutput, it is
recommended not doing so. Instead, emit one record into the ReaderOutput and return a InputStatus.MORE_AVAILABLE
to let the caller thread know there are more records available.
pollNext
in interface SourceReader<O,SplitT extends IteratorSourceSplit<E,IterT>>
protected void finishSplit()
protected InputStatus tryMoveToNextSplit()
public CompletableFuture<Void> isAvailable()
SourceReader
Once the future completes, the runtime will keep calling the SourceReader.pollNext(ReaderOutput)
method until that method returns a status other than InputStatus.MORE_AVAILABLE
. After that, the runtime will again call this method to obtain
the next future. Once that completes, it will again call SourceReader.pollNext(ReaderOutput)
and
so on.
The contract is the following: If the reader has data available, then all futures previously returned by this method must eventually complete. Otherwise the source might stall indefinitely.
It is not a problem to have occasional "false positives", meaning to complete a future
even if no data is available. However, one should not use an "always complete" future in
cases no data is available, because that will result in busy waiting loops calling pollNext(...)
even though no data is available.
isAvailable
in interface SourceReader<O,SplitT extends IteratorSourceSplit<E,IterT>>
public void addSplits(List<SplitT> splits)
SourceReader
SplitEnumeratorContext.assignSplit(SourceSplit, int)
or SplitEnumeratorContext.assignSplits(SplitsAssignment)
.addSplits
in interface SourceReader<O,SplitT extends IteratorSourceSplit<E,IterT>>
splits
- The splits assigned by the split enumerator.public void notifyNoMoreSplits()
SourceReader
It is triggered when the enumerator calls SplitEnumeratorContext.signalNoMoreSplits(int)
with the reader's parallel subtask.
notifyNoMoreSplits
in interface SourceReader<O,SplitT extends IteratorSourceSplit<E,IterT>>
public List<SplitT> snapshotState(long checkpointId)
SourceReader
snapshotState
in interface SourceReader<O,SplitT extends IteratorSourceSplit<E,IterT>>
public void close() throws Exception
close
in interface AutoCloseable
Exception
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.