org.apache.flink.api.connector.source

Interface SplitEnumerator<SplitT extends SourceSplit,CheckpointT>

All Superinterfaces:

AutoCloseable, CheckpointListener

All Known Implementing Classes:

ContinuousFileSplitEnumerator, ContinuousHiveSplitEnumerator, IteratorSourceEnumerator, KafkaSourceEnumerator, StaticFileSplitEnumerator

@PublicEvolving
public interface SplitEnumerator<SplitT extends SourceSplit,CheckpointT>
extends AutoCloseable, CheckpointListener

A interface of a split enumerator responsible for the followings: 1. discover the splits for the SourceReader to read. 2. assign the splits to the source reader.

Method Summary

Modifier and Type	Method and Description
void	addReader(int subtaskId) Add a new source reader with the given subtask ID.
void	addSplitsBack(List<SplitT> splits, int subtaskId) Add a split back to the split enumerator.
void	close() Called to close the enumerator, in case it holds on to any resources, like threads or network connections.
default void	handleSourceEvent(int subtaskId, SourceEvent sourceEvent) Handles a custom source event from the source reader.
void	handleSplitRequest(int subtaskId, String requesterHostname) Handles the request for a split.
default void	notifyCheckpointComplete(long checkpointId) We have an empty default implementation here because most source readers do not have to implement the method.
CheckpointT	snapshotState() Checkpoints the state of this split enumerator.
void	start() Start the split enumerator.

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted

Method Detail

start

void start()

Start the split enumerator.

The default behavior does nothing.

handleSplitRequest

void handleSplitRequest(int subtaskId,
                        @Nullable
                        String requesterHostname)

Handles the request for a split. This method is called when the reader with the given subtask id calls the SourceReaderContext.sendSplitRequest() method.

Parameters:

subtaskId - the subtask id of the source reader who sent the source event.

requesterHostname - Optional, the hostname where the requesting task is running. This can be used to make split assignments locality-aware.

addSplitsBack

void addSplitsBack(List<SplitT> splits,
                   int subtaskId)

Add a split back to the split enumerator. It will only happen when a SourceReader fails and there are splits assigned to it after the last successful checkpoint.

Parameters:

splits - The split to add back to the enumerator for reassignment.

subtaskId - The id of the subtask to which the returned splits belong.

addReader

void addReader(int subtaskId)

Add a new source reader with the given subtask ID.

Parameters:

subtaskId - the subtask ID of the new source reader.

snapshotState

CheckpointT snapshotState()
                   throws Exception

Checkpoints the state of this split enumerator.

Returns:

an object containing the state of the split enumerator.

Throws:

Exception - when the snapshot cannot be taken.

close

void close()
    throws IOException

Called to close the enumerator, in case it holds on to any resources, like threads or network connections.

Specified by:

close in interface AutoCloseable

Throws:

IOException

notifyCheckpointComplete

default void notifyCheckpointComplete(long checkpointId)
                               throws Exception

We have an empty default implementation here because most source readers do not have to implement the method.

Specified by:

notifyCheckpointComplete in interface CheckpointListener

Parameters:

checkpointId - The ID of the checkpoint that has been completed.

Throws:

Exception - This method can propagate exceptions, which leads to a failure/recovery for the task. Not that this will NOT lead to the checkpoint being revoked.

See Also:

CheckpointListener.notifyCheckpointComplete(long)

handleSourceEvent

default void handleSourceEvent(int subtaskId,
                               SourceEvent sourceEvent)

Handles a custom source event from the source reader.

This method has a default implementation that does nothing, because it is only required to be implemented by some sources, which have a custom event protocol between reader and enumerator. The common events for reader registration and split requests are not dispatched to this method, but rather invoke the addReader(int) and handleSplitRequest(int, String) methods.

Parameters:

subtaskId - the subtask id of the source reader who sent the source event.

sourceEvent - the source event from the source reader.