Package org.apache.flink.api.connector.source

Interface ExternallyInducedSourceReader<T,​SplitT extends SourceSplit>

  • Type Parameters:
    T - The type of records produced by the source.
    SplitT - The type of splits handled by the source.
    All Superinterfaces:
    AutoCloseable, CheckpointListener, SourceReader<T,​SplitT>
    @Experimental
@PublicEvolving
public interface ExternallyInducedSourceReader<T,​SplitT extends SourceSplit>
extends SourceReader<T,​SplitT>
    Sources that implement this interface delay checkpoints when receiving a trigger message from the checkpoint coordinator to the point when their input data/events indicate that a checkpoint should be triggered.

    The ExternallyInducedSourceReader tells the Flink runtime that a checkpoint needs to be made by returning a checkpointId when shouldTriggerCheckpoint() is invoked.

    The implementations typically works together with the SplitEnumerator which informs the external system to trigger a checkpoint. The external system also needs to forward the Checkpoint ID to the source, so the source knows which checkpoint to trigger.

    Important: It is crucial that all parallel source tasks trigger their checkpoints at roughly the same time. Otherwise this leads to performance issues due to long checkpoint alignment phases or large alignment data snapshots.

    • Method Detail

      • shouldTriggerCheckpoint

        Optional<Long> shouldTriggerCheckpoint()
        A method that informs the Flink runtime whether a checkpoint should be triggered on this Source.

        This method is invoked when the previous SourceReader.pollNext(ReaderOutput) returns InputStatus.NOTHING_AVAILABLE, to check if the source needs to be checkpointed.

        If a CheckpointId is returned, a checkpoint will be triggered on this source reader. Otherwise, Flink runtime will continue to process the records.

        Returns:
        An optional checkpoint ID that Flink runtime should take a checkpoint for.