@Internal public final class WatermarkStatus extends StreamElement
IDLE
and ACTIVE
. Watermark Status elements are generated
at the sources, and may be propagated through the tasks of the topology. They directly infer the
current status of the emitting task; a SourceStreamTask
or StreamTask
emits a
IDLE
if it will temporarily halt to emit any watermarks (i.e. is idle),
and emits a ACTIVE
once it resumes to do so (i.e. is active). Tasks are
responsible for propagating their status further downstream once they toggle between being idle
and active. The cases that source tasks and downstream tasks are considered either idle or active
is explained below:
StreamSource
, will not emit watermarks for an indefinite amount of time. This is the case,
for example, for Flink's Kafka Consumer, where sources might initially have no assigned
partitions to read from, or no records can be read from the assigned partitions. Once the
head StreamSource
operator detects that it will resume emitting data, the source
task is considered to be active. StreamSource
s are responsible for toggling the
status of the containing source task and ensuring that no watermarks will be emitted while
the task is idle. This guarantee should be enforced on sources through SourceFunction.SourceContext
implementations.
IDLE
. As long as one of its input streams is active, i.e. the last
received Watermark Status element from the input stream is ACTIVE
,
the task is active.
Watermark Status elements received at downstream tasks also affect and control how their
operators process and advance their watermarks. The below describes the effects (the logic is
implemented as a StatusWatermarkValve
which downstream tasks should use for such
purposes):
Note that to notify downstream tasks that a source task is permanently closed and will no
longer send any more elements, the source should still send a Watermark.MAX_WATERMARK
instead of IDLE
. Watermark Status elements only serve as markers for
temporary status.
Modifier and Type | Field and Description |
---|---|
static WatermarkStatus |
ACTIVE |
static int |
ACTIVE_STATUS |
static WatermarkStatus |
IDLE |
static int |
IDLE_STATUS |
int |
status |
Constructor and Description |
---|
WatermarkStatus(int status) |
Modifier and Type | Method and Description |
---|---|
boolean |
equals(Object o) |
int |
getStatus() |
int |
hashCode() |
boolean |
isActive() |
boolean |
isIdle() |
String |
toString() |
asLatencyMarker, asRecord, asWatermark, asWatermarkStatus, isLatencyMarker, isRecord, isWatermark, isWatermarkStatus
public static final int IDLE_STATUS
public static final int ACTIVE_STATUS
public static final WatermarkStatus IDLE
public static final WatermarkStatus ACTIVE
public final int status
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.