org.apache.flink.api.connector.source

Interface ExternallyInducedSourceReader<T,SplitT extends SourceSplit>

Type Parameters:

T - The type of records produced by the source.

SplitT - The type of splits handled by the source.

All Superinterfaces:

AutoCloseable, CheckpointListener, SourceReader<T,SplitT>

@Experimental
 @PublicEvolving
public interface ExternallyInducedSourceReader<T,SplitT extends SourceSplit>
extends SourceReader<T,SplitT>

Sources that implement this interface delay checkpoints when receiving a trigger message from the checkpoint coordinator to the point when their input data/events indicate that a checkpoint should be triggered.

The ExternallyInducedSourceReader tells the Flink runtime that a checkpoint needs to be made by returning a checkpointId when shouldTriggerCheckpoint() is invoked.

The implementations typically works together with the SplitEnumerator which informs the external system to trigger a checkpoint. The external system also needs to forward the Checkpoint ID to the source, so the source knows which checkpoint to trigger.

Important: It is crucial that all parallel source tasks trigger their checkpoints at roughly the same time. Otherwise this leads to performance issues due to long checkpoint alignment phases or large alignment data snapshots.

Method Summary

Modifier and Type	Method and Description
Optional<Long>	shouldTriggerCheckpoint() A method that informs the Flink runtime whether a checkpoint should be triggered on this Source.

Methods inherited from interface org.apache.flink.api.connector.source.SourceReader

addSplits, handleSourceEvents, isAvailable, notifyCheckpointComplete, notifyNoMoreSplits, pauseOrResumeSplits, pollNext, snapshotState, start

Methods inherited from interface java.lang.AutoCloseable

close

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted

Method Detail

shouldTriggerCheckpoint

Optional<Long> shouldTriggerCheckpoint()

A method that informs the Flink runtime whether a checkpoint should be triggered on this Source.

This method is invoked when the previous SourceReader.pollNext(ReaderOutput) returns InputStatus.NOTHING_AVAILABLE, to check if the source needs to be checkpointed.

If a CheckpointId is returned, a checkpoint will be triggered on this source reader. Otherwise, Flink runtime will continue to process the records.

Returns:

An optional checkpoint ID that Flink runtime should take a checkpoint for.