IN
- Input type for the operator.OUT
- Output type for the operator.@Internal public class AsyncWaitOperator<IN,OUT> extends AbstractUdfStreamOperator<OUT,AsyncFunction<IN,OUT>> implements OneInputStreamOperator<IN,OUT>, BoundedOneInput
AsyncWaitOperator
allows to asynchronously process incoming stream records. For that
the operator creates an ResultFuture
which is passed to an AsyncFunction
. Within
the async function, the user can complete the async collector arbitrarily. Once the async
collector has been completed, the result is emitted by the operator's emitter to downstream
operators.
The operator offers different output modes depending on the chosen AsyncDataStream.OutputMode
. In
order to give exactly once processing guarantees, the operator stores all currently in-flight
StreamElement
in it's operator state. Upon recovery the recorded set of stream elements
is replayed.
The operator also support retry on predefined condition and strategy AsyncRetryStrategy
. If given strategy is NO_RETRY_STRATEGY or a custom implemented AsyncRetryPredicate
which defines empty retry
predicate, then no retry will happen, everything keeps the same as the AsyncWaitOperator
without retry support.
In case of chaining of this operator, it has to be made sure that the operators in the chain
are opened tail to head. The reason for this is that an opened AsyncWaitOperator
starts
already emitting recovered StreamElement
to downstream operators.
userFunction
chainingStrategy, config, lastRecordAttributes1, lastRecordAttributes2, latencyStats, LOG, metrics, output, processingTimeService, stateHandler, stateKeySelector1, stateKeySelector2, timeServiceManager
Constructor and Description |
---|
AsyncWaitOperator(AsyncFunction<IN,OUT> asyncFunction,
long timeout,
int capacity,
AsyncDataStream.OutputMode outputMode,
AsyncRetryStrategy<OUT> asyncRetryStrategy,
ProcessingTimeService processingTimeService,
MailboxExecutor mailboxExecutor) |
Modifier and Type | Method and Description |
---|---|
void |
endInput()
It is notified that no more data will arrive from the input.
|
void |
initializeState(StateInitializationContext context)
Stream operators with state which can be restored need to override this hook method.
|
void |
open()
This method is called immediately before any elements are processed, it should contain the
operator's initialization logic, e.g. state initialization.
|
void |
processElement(StreamRecord<IN> record)
Processes one element that arrived on this input of the
MultipleInputStreamOperator . |
void |
processWatermark(Watermark mark)
Processes a
Watermark that arrived on the first input of this two-input operator. |
void |
setup(StreamTask<?,?> containingTask,
StreamConfig config,
Output<StreamRecord<OUT>> output)
Initializes the operator.
|
void |
snapshotState(StateSnapshotContext context)
Stream operators with state, which want to participate in a snapshot need to override this
hook method.
|
close, finish, getUserFunction, getUserFunctionParameters, notifyCheckpointAborted, notifyCheckpointComplete, setOutputType
getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, isUsingCustomRawKeyedState, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setMailboxExecutor, setProcessingTimeService, snapshotState, useSplittableTimers
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
setKeyContextElement
close, finish, getMetricGroup, getOperatorAttributes, getOperatorID, initializeState, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState
notifyCheckpointAborted, notifyCheckpointComplete
getCurrentKey, setCurrentKey
processLatencyMarker, processRecordAttributes, processWatermarkStatus
hasKeyContext
public AsyncWaitOperator(@Nonnull AsyncFunction<IN,OUT> asyncFunction, long timeout, int capacity, @Nonnull AsyncDataStream.OutputMode outputMode, @Nonnull AsyncRetryStrategy<OUT> asyncRetryStrategy, @Nonnull ProcessingTimeService processingTimeService, @Nonnull MailboxExecutor mailboxExecutor)
public void setup(StreamTask<?,?> containingTask, StreamConfig config, Output<StreamRecord<OUT>> output)
SetupableStreamOperator
setup
in interface SetupableStreamOperator<OUT>
setup
in class AbstractUdfStreamOperator<OUT,AsyncFunction<IN,OUT>>
public void open() throws Exception
AbstractStreamOperator
The default implementation does nothing.
open
in interface StreamOperator<OUT>
open
in class AbstractUdfStreamOperator<OUT,AsyncFunction<IN,OUT>>
Exception
- An exception in this method causes the operator to fail.public void processElement(StreamRecord<IN> record) throws Exception
Input
MultipleInputStreamOperator
.
This method is guaranteed to not be called concurrently with other methods of the operator.processElement
in interface Input<IN>
Exception
public void processWatermark(Watermark mark) throws Exception
Input
Watermark
that arrived on the first input of this two-input operator.
This method is guaranteed to not be called concurrently with other methods of the operator.processWatermark
in interface Input<IN>
processWatermark
in class AbstractStreamOperator<OUT>
Exception
Watermark
public void snapshotState(StateSnapshotContext context) throws Exception
AbstractStreamOperator
snapshotState
in interface StreamOperatorStateHandler.CheckpointedStreamOperator
snapshotState
in class AbstractUdfStreamOperator<OUT,AsyncFunction<IN,OUT>>
context
- context that provides information and means required for taking a snapshotException
public void initializeState(StateInitializationContext context) throws Exception
AbstractStreamOperator
initializeState
in interface StreamOperatorStateHandler.CheckpointedStreamOperator
initializeState
in class AbstractUdfStreamOperator<OUT,AsyncFunction<IN,OUT>>
context
- context that allows to register different states.Exception
public void endInput() throws Exception
BoundedOneInput
WARNING: It is not safe to use this method to commit any transactions or other side
effects! You can use this method to flush any buffered data that can later on be committed
e.g. in a CheckpointListener.notifyCheckpointComplete(long)
.
NOTE: Given it is semantically very similar to the StreamOperator.finish()
method. It might be dropped in favour of the other method at some point in time.
endInput
in interface BoundedOneInput
Exception
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.