Interface FileSplitAssigner
-
- All Known Implementing Classes:
LocalityAwareSplitAssigner
,SimpleSplitAssigner
@PublicEvolving public interface FileSplitAssigner
TheFileSplitAssigner
is responsible for deciding what split should be processed next by which node. It determines split processing order and locality.
-
-
Nested Class Summary
Nested Classes Modifier and Type Interface Description static interface
FileSplitAssigner.Provider
Factory for theFileSplitAssigner
, to allow theFileSplitAssigner
to be eagerly initialized and to not be serializable.
-
Method Summary
All Methods Instance Methods Abstract Methods Modifier and Type Method Description void
addSplits(Collection<FileSourceSplit> splits)
Adds a set of splits to this assigner.Optional<FileSourceSplit>
getNext(String hostname)
Gets the next split.Collection<FileSourceSplit>
remainingSplits()
Gets the remaining splits that this assigner has pending.
-
-
-
Method Detail
-
getNext
Optional<FileSourceSplit> getNext(@Nullable String hostname)
Gets the next split.When this method returns an empty
Optional
, then the set of splits is assumed to be done and the source will finish once the readers finished their current splits.
-
addSplits
void addSplits(Collection<FileSourceSplit> splits)
Adds a set of splits to this assigner. This happens for example when some split processing failed and the splits need to be re-added, or when new splits got discovered.
-
remainingSplits
Collection<FileSourceSplit> remainingSplits()
Gets the remaining splits that this assigner has pending.
-
-