org.apache.flink.connector.base.source.reader.splitreader

Interface SplitReader<E,SplitT extends SourceSplit>

Type Parameters:

E - the element type.

SplitT - the split type.

All Known Implementing Classes:

KafkaPartitionSplitReader

@PublicEvolving
public interface SplitReader<E,SplitT extends SourceSplit>

An interface used to read from splits. The implementation could either read from a single split or from multiple splits.

Method Summary

Modifier and Type	Method and Description
void	close() Close the split reader.
RecordsWithSplitIds<E>	fetch() Fetch elements into the blocking queue for the given splits.
void	handleSplitsChanges(SplitsChange<SplitT> splitsChanges) Handle the split changes.
default void	pauseOrResumeSplits(Collection<SplitT> splitsToPause, Collection<SplitT> splitsToResume) Pauses or resumes reading of individual splits readers.
void	wakeUp() Wake up the split reader in case the fetcher thread is blocking in fetch().

Method Detail

fetch

RecordsWithSplitIds<E> fetch()
                      throws IOException

Fetch elements into the blocking queue for the given splits. The fetch call could be blocking but it should get unblocked when wakeUp() is invoked. In that case, the implementation may either decide to return without throwing an exception, or it can just throw an interrupted exception. In either case, this method should be reentrant, meaning that the next fetch call should just resume from where the last fetch call was waken up or interrupted.

Returns:

the Ids of the finished splits.

Throws:

IOException - when encountered IO errors, such as deserialization failures.

handleSplitsChanges

void handleSplitsChanges(SplitsChange<SplitT> splitsChanges)

Handle the split changes. This call should be non-blocking.

For the consistency of internal state in SourceReaderBase, if an invalid split is added to the reader (for example splits without any records), it should be put back into RecordsWithSplitIds as finished splits so that SourceReaderBase could be able to clean up resources created for it.

Parameters:

splitsChanges - the split changes that the SplitReader needs to handle.

wakeUp

void wakeUp()

Wake up the split reader in case the fetcher thread is blocking in fetch().

close

void close()
    throws Exception

Close the split reader.

Throws:

Exception - if closing the split reader failed.

pauseOrResumeSplits

default void pauseOrResumeSplits(Collection<SplitT> splitsToPause,
                                 Collection<SplitT> splitsToResume)

Pauses or resumes reading of individual splits readers.

Note that no other methods can be called in parallel, so it's fine to non-atomically update subscriptions. This method is simply providing connectors with more expressive APIs the opportunity to update all subscriptions at once.

This is currently used to align the watermarks of splits, if watermark alignment is used and the source reads from more than one split.

The default implementation throws an UnsupportedOperationException where the default implementation will be removed in future releases. To be compatible with future releases, it is recommended to implement this method and override the default implementation.

Parameters:

splitsToPause - the splits to pause

splitsToResume - the splits to resume