org.apache.flink.table.connector.source.abilities

Interface SupportsWatermarkPushDown

All Known Implementing Classes:

KafkaDynamicSource

@PublicEvolving
public interface SupportsWatermarkPushDown

Enables to push down a watermark strategy into a ScanTableSource.

The concept of watermarks defines when time operations based on an event time attribute will be triggered. A watermark tells operators that no elements with a timestamp older or equal to the watermark timestamp should arrive at the operator. Thus, watermarks are a trade-off between latency and completeness.

Given the following SQL:

 CREATE TABLE t (i INT, ts TIMESTAMP(3), WATERMARK FOR ts AS ts - INTERVAL '5' SECOND)  // `ts` becomes a time attribute
 

In the above example, generated watermarks are lagging 5 seconds behind the highest seen timestamp.

For correctness, it might be necessary to perform the watermark generation as early as possible in order to be close to the actual data generation within a source's data partition.

If the ScanTableSource.getScanRuntimeProvider(ScanTableSource.ScanContext) returns SourceProvider, watermarks will be automatically pushed into the runtime source operator by the framework. In this case, this interface does not need to be implemented.

If the ScanTableSource does not return a SourceProvider and this interface is not implemented, watermarks are generated in a subsequent operation after the source. In this case, it is recommended to implement this interface to perform the watermark generation within source's data partition.

This interface provides a WatermarkStrategy that needs to be applied to the runtime implementation. Most built-in Flink sources provide a way of setting the watermark generator.

See Also:

SupportsSourceWatermark

Method Summary

Modifier and Type	Method and Description
void	applyWatermark(WatermarkStrategy<RowData> watermarkStrategy) Provides a WatermarkStrategy which defines how to generate Watermarks in the stream source.

Method Detail

applyWatermark

void applyWatermark(WatermarkStrategy<RowData> watermarkStrategy)

Provides a WatermarkStrategy which defines how to generate Watermarks in the stream source.

The WatermarkStrategy is a builder/factory for the actual runtime implementation consisting of TimestampAssigner (assigns the event-time timestamps to each record) and the WatermarkGenerator (generates the watermarks).

Note: If necessary, the watermark strategy will contain required computed column expressions and consider metadata columns (if SupportsReadingMetadata is implemented).