@Internal public class StaticFileSplitEnumerator extends Object implements SplitEnumerator<FileSourceSplit,PendingSplitsCheckpoint<FileSourceSplit>>
FileSource
input.
This enumerator takes all files that are present in the configured input directories and assigns them to the readers. Once all files are processed, the source is finished.
The implementation of this class is rather thin. The actual logic for creating the set of
FileSourceSplits to process, and the logic to decide which reader gets what split, are in FileEnumerator
and in FileSplitAssigner
, respectively.
Constructor and Description |
---|
StaticFileSplitEnumerator(SplitEnumeratorContext<FileSourceSplit> context,
FileSplitAssigner splitAssigner) |
Modifier and Type | Method and Description |
---|---|
void |
addReader(int subtaskId)
Add a new source reader with the given subtask ID.
|
void |
addSplitsBack(List<FileSourceSplit> splits,
int subtaskId)
Add a split back to the split enumerator.
|
void |
close()
Called to close the enumerator, in case it holds on to any resources, like threads or network
connections.
|
void |
handleSourceEvent(int subtaskId,
SourceEvent sourceEvent)
Handles a custom source event from the source reader.
|
void |
handleSplitRequest(int subtask,
String hostname)
Handles the request for a split.
|
PendingSplitsCheckpoint<FileSourceSplit> |
snapshotState()
Checkpoints the state of this split enumerator.
|
void |
start()
Start the split enumerator.
|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
notifyCheckpointComplete
notifyCheckpointAborted
public StaticFileSplitEnumerator(SplitEnumeratorContext<FileSourceSplit> context, FileSplitAssigner splitAssigner)
public void start()
SplitEnumerator
The default behavior does nothing.
start
in interface SplitEnumerator<FileSourceSplit,PendingSplitsCheckpoint<FileSourceSplit>>
public void close() throws IOException
SplitEnumerator
close
in interface AutoCloseable
close
in interface SplitEnumerator<FileSourceSplit,PendingSplitsCheckpoint<FileSourceSplit>>
IOException
public void addReader(int subtaskId)
SplitEnumerator
addReader
in interface SplitEnumerator<FileSourceSplit,PendingSplitsCheckpoint<FileSourceSplit>>
subtaskId
- the subtask ID of the new source reader.public void handleSplitRequest(int subtask, @Nullable String hostname)
SplitEnumerator
SourceReaderContext.sendSplitRequest()
method.handleSplitRequest
in interface SplitEnumerator<FileSourceSplit,PendingSplitsCheckpoint<FileSourceSplit>>
subtask
- the subtask id of the source reader who sent the source event.hostname
- Optional, the hostname where the requesting task is running. This
can be used to make split assignments locality-aware.public void handleSourceEvent(int subtaskId, SourceEvent sourceEvent)
SplitEnumerator
This method has a default implementation that does nothing, because it is only required to
be implemented by some sources, which have a custom event protocol between reader and
enumerator. The common events for reader registration and split requests are not dispatched
to this method, but rather invoke the SplitEnumerator.addReader(int)
and SplitEnumerator.handleSplitRequest(int, String)
methods.
handleSourceEvent
in interface SplitEnumerator<FileSourceSplit,PendingSplitsCheckpoint<FileSourceSplit>>
subtaskId
- the subtask id of the source reader who sent the source event.sourceEvent
- the source event from the source reader.public void addSplitsBack(List<FileSourceSplit> splits, int subtaskId)
SplitEnumerator
SourceReader
fails and there are splits assigned to it after the last successful checkpoint.addSplitsBack
in interface SplitEnumerator<FileSourceSplit,PendingSplitsCheckpoint<FileSourceSplit>>
splits
- The split to add back to the enumerator for reassignment.subtaskId
- The id of the subtask to which the returned splits belong.public PendingSplitsCheckpoint<FileSourceSplit> snapshotState()
SplitEnumerator
snapshotState
in interface SplitEnumerator<FileSourceSplit,PendingSplitsCheckpoint<FileSourceSplit>>
Copyright © 2014–2021 The Apache Software Foundation. All rights reserved.