@PublicEvolving public interface SupportsWatermarkPushDown
The concept of watermarks defines when time operations based on an event time attribute will be triggered. A watermark tells operators that no elements with a timestamp older or equal to the watermark timestamp should arrive at the operator. Thus, watermarks are a trade-off between latency and completeness.
Given the following SQL:
CREATE TABLE t (i INT, ts TIMESTAMP(3), WATERMARK FOR ts AS ts - INTERVAL '5' SECOND) // `ts` becomes a time attribute
In the above example, generated watermarks are lagging 5 seconds behind the highest seen timestamp.
For correctness, it might be necessary to perform the watermark generation as early as possible in order to be close to the actual data generation within a source's data partition.
SourceProvider, watermarks will be automatically pushed into the runtime source operator
by the framework. In this case, this interface does not need to be implemented.
ScanTableSource does not return a
SourceProvider and this interface is
not implemented, watermarks are generated in a subsequent operation after the source. In this
case, it is recommended to implement this interface to perform the watermark generation within
source's data partition.
This interface provides a
WatermarkStrategy that needs to be applied to the runtime
implementation. Most built-in Flink sources provide a way of setting the watermark generator.
void applyWatermark(WatermarkStrategy<RowData> watermarkStrategy)
WatermarkStrategy which defines how to generate
Watermarks in the
WatermarkStrategy is a builder/factory for the actual runtime implementation
TimestampAssigner (assigns the event-time timestamps to each record)
WatermarkGenerator (generates the watermarks).
Note: If necessary, the watermark strategy will contain required computed column
expressions and consider metadata columns (if
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.