org.apache.flink.api.connector.source

Interface SplitEnumeratorContext<SplitT extends SourceSplit>

Type Parameters:

SplitT - the type of the splits.

All Known Implementing Classes:

SourceCoordinatorContext, TestingSplitEnumeratorContext

@PublicEvolving
public interface SplitEnumeratorContext<SplitT extends SourceSplit>

A context class for the SplitEnumerator. This class serves the following purposes: 1. Host information necessary for the SplitEnumerator to make split assignment decisions. 2. Accept and track the split assignment from the enumerator. 3. Provide a managed threading model so the split enumerators do not need to create their own internal threads.

Method Summary

Modifier and Type	Method and Description
default void	assignSplit(SplitT split, int subtask) Assigns a single split.
void	assignSplits(SplitsAssignment<SplitT> newSplitAssignments) Assign the splits.
<T> void	callAsync(Callable<T> callable, java.util.function.BiConsumer<T,Throwable> handler) Invoke the callable and handover the return value to the handler which will be executed by the source coordinator.
<T> void	callAsync(Callable<T> callable, java.util.function.BiConsumer<T,Throwable> handler, long initialDelay, long period) Invoke the given callable periodically and handover the return value to the handler which will be executed by the source coordinator.
int	currentParallelism() Get the current parallelism of this Source.
MetricGroup	metricGroup()
Map<Integer,ReaderInfo>	registeredReaders() Get the currently registered readers.
void	runInCoordinatorThread(Runnable runnable) Invoke the given runnable in the source coordinator thread.
void	sendEventToSourceReader(int subtaskId, SourceEvent event) Send a source event to a source reader.
void	signalNoMoreSplits(int subtask) Signals a subtask that it will not receive any further split.

Method Detail

metricGroup

MetricGroup metricGroup()

sendEventToSourceReader

void sendEventToSourceReader(int subtaskId,
                             SourceEvent event)

Send a source event to a source reader. The source reader is identified by its subtask id.

Parameters:

subtaskId - the subtask id of the source reader to send this event to.

event - the source event to send.

currentParallelism

int currentParallelism()

Get the current parallelism of this Source. Note that due to auto-scaling, the parallelism may change over time. Therefore the SplitEnumerator should not cache the return value of this method, but always invoke this method to get the latest parallelism.

Returns:

the parallelism of the Source.

registeredReaders

Map<Integer,ReaderInfo> registeredReaders()

Get the currently registered readers. The mapping is from subtask id to the reader info.

Returns:

the currently registered readers.

assignSplits

void assignSplits(SplitsAssignment<SplitT> newSplitAssignments)

Assign the splits.

Parameters:

newSplitAssignments - the new split assignments to add.

assignSplit

default void assignSplit(SplitT split,
                         int subtask)

Assigns a single split.

When assigning multiple splits, it is more efficient to assign all of them in a single call to the assignSplits(SplitsAssignment) method.

Parameters:

split - The new split

subtask - The index of the operator's parallel subtask that shall receive the split.

signalNoMoreSplits

void signalNoMoreSplits(int subtask)

Signals a subtask that it will not receive any further split.

Parameters:

subtask - The index of the operator's parallel subtask that shall be signaled it will not receive any further split.

callAsync

<T> void callAsync(Callable<T> callable,
                   java.util.function.BiConsumer<T,Throwable> handler)

Invoke the callable and handover the return value to the handler which will be executed by the source coordinator. When this method is invoked multiple times, The Callable s may be executed in a thread pool concurrently.

It is important to make sure that the callable does not modify any shared state, especially the states that will be a part of the SplitEnumerator.snapshotState(). Otherwise the there might be unexpected behavior.

Note that an exception thrown from the handler would result in failing the job.

Parameters:

callable - a callable to call.

handler - a handler that handles the return value of or the exception thrown from the callable.

callAsync

<T> void callAsync(Callable<T> callable,
                   java.util.function.BiConsumer<T,Throwable> handler,
                   long initialDelay,
                   long period)

Invoke the given callable periodically and handover the return value to the handler which will be executed by the source coordinator. When this method is invoked multiple times, The Callables may be executed in a thread pool concurrently.

It is important to make sure that the callable does not modify any shared state, especially the states that will be a part of the SplitEnumerator.snapshotState(). Otherwise the there might be unexpected behavior.

Note that an exception thrown from the handler would result in failing the job.

Parameters:

callable - the callable to call.

handler - a handler that handles the return value of or the exception thrown from the callable.

initialDelay - the initial delay of calling the callable.

period - the period between two invocations of the callable.

runInCoordinatorThread

void runInCoordinatorThread(Runnable runnable)

Invoke the given runnable in the source coordinator thread.

This can be useful when the enumerator needs to execute some action (like assignSplits) triggered by some external events. E.g., Watermark from another source advanced and this source now be able to assign splits to awaiting readers. The trigger can be initiated from the coordinator thread of the other source. Instead of using lock for thread safety, this API allows to run such externally triggered action in the coordinator thread. Hence, we can ensure all enumerator actions are serialized in the single coordinator thread.

It is important that the runnable does not block.

Parameters:

runnable - a runnable to execute