@PublicEvolving public interface SupportsWatermarkPushDown
The concept of watermarks defines when time operations based on an event time attribute will be triggered. A watermark tells operators that no elements with a timestamp older or equal to the watermark timestamp should arrive at the operator. Thus, watermarks are a trade-off between latency and completeness.
Given the following SQL:
CREATE TABLE t (i INT, ts TIMESTAMP(3), WATERMARK FOR ts AS ts - INTERVAL '5' SECOND) // `ts` becomes a time attribute
In the above example, generated watermarks are lagging 5 seconds behind the highest seen timestamp.
By default, if this interface is not implemented, watermarks are generated in a subsequent operation after the source.
However, for correctness, it might be necessary to perform the watermark generation as early as possible in order to be close to the actual data generation within a source's data partition.
This interface provides a
SupportsWatermarkPushDown.WatermarkProvider that needs to be applied to a runtime
implementation. Most built-in Flink sources provide a way of setting the watermark generator.
Note: In many cases, watermarks are generated from computed columns. If a source implements
this interface, it is recommended to also implement
|Modifier and Type||Interface and Description|
Provides actual runtime implementation for generating watermarks.
void applyWatermark(SupportsWatermarkPushDown.WatermarkProvider provider)
There exist different interfaces for runtime implementation which is why
SupportsWatermarkPushDown.WatermarkProvider serves as the base interface. Concrete
interfaces might be located in other Flink modules.
Implementations need to perform an
instanceof check and fail with an exception if
SupportsWatermarkPushDown.WatermarkProvider is unsupported.
Copyright © 2014–2021 The Apache Software Foundation. All rights reserved.