org.apache.flink.state.api.output.operators

Class StateBootstrapWrapperOperator<IN,OUT,OP extends AbstractStreamOperator<OUT> & OneInputStreamOperator<IN,OUT>>

java.lang.Object

org.apache.flink.state.api.output.operators.StateBootstrapWrapperOperator<IN,OUT,OP>

All Implemented Interfaces:

Serializable, CheckpointListener, BoundedOneInput, Input<IN>, KeyContext, OneInputStreamOperator<IN,TaggedOperatorSubtaskState>, SetupableStreamOperator<TaggedOperatorSubtaskState>, StreamOperator<TaggedOperatorSubtaskState>

@Internal
public final class StateBootstrapWrapperOperator<IN,OUT,OP extends AbstractStreamOperator<OUT> & OneInputStreamOperator<IN,OUT>>
extends Object
implements OneInputStreamOperator<IN,TaggedOperatorSubtaskState>, SetupableStreamOperator<TaggedOperatorSubtaskState>, BoundedOneInput

Wraps an existing operator so it can be bootstrapped.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
StateBootstrapWrapperOperator(long timestamp, Path savepointPath, OP operator)

Method Summary

Modifier and Type	Method and Description
void	close() This method is called at the very end of the operator's life, both in the case of a successful completion of the operation, and in the case of a failure and canceling.
void	endInput() It is notified that no more data will arrive from the input.
void	finish() This method is called at the end of data processing.
ChainingStrategy	getChainingStrategy()
Object	getCurrentKey()
OperatorMetricGroup	getMetricGroup()
OperatorID	getOperatorID()
void	initializeState(StreamTaskStateInitializer streamTaskStateManager) Provides a context to initialize all state in the operator.
void	notifyCheckpointComplete(long checkpointId) Notifies the listener that the checkpoint with the given checkpointId completed and was committed.
void	open() This method is called immediately before any elements are processed, it should contain the operator's initialization logic.
void	prepareSnapshotPreBarrier(long checkpointId) This method is called when the operator should do a snapshot, before it emits its own checkpoint barrier.
void	processElement(StreamRecord<IN> element) Processes one element that arrived on this input of the MultipleInputStreamOperator.
void	processLatencyMarker(LatencyMarker latencyMarker) Processes a LatencyMarker that arrived on the first input of this two-input operator.
void	processWatermark(Watermark mark) Processes a Watermark that arrived on the first input of this two-input operator.
void	processWatermarkStatus(WatermarkStatus watermarkStatus) Processes a WatermarkStatus that arrived on this input of the MultipleInputStreamOperator.
void	setChainingStrategy(ChainingStrategy strategy)
void	setCurrentKey(Object key)
void	setKeyContextElement1(StreamRecord<?> record)
void	setKeyContextElement2(StreamRecord<?> record)
void	setup(StreamTask<?,?> containingTask, StreamConfig config, Output<StreamRecord<TaggedOperatorSubtaskState>> output) Initializes the operator.
OperatorSnapshotFutures	snapshotState(long checkpointId, long timestamp, CheckpointOptions checkpointOptions, CheckpointStreamFactory storageLocation) Called to draw a state snapshot from the operator.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

setKeyContextElement

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted

Constructor Detail

StateBootstrapWrapperOperator

public StateBootstrapWrapperOperator(long timestamp,
                                     Path savepointPath,
                                     OP operator)

Method Detail

setup

public void setup(StreamTask<?,?> containingTask,
                  StreamConfig config,
                  Output<StreamRecord<TaggedOperatorSubtaskState>> output)

Description copied from interface: SetupableStreamOperator

Initializes the operator. Sets access to the context and the output.

Specified by:

setup in interface SetupableStreamOperator<TaggedOperatorSubtaskState>

processElement

public void processElement(StreamRecord<IN> element)
                    throws Exception

Description copied from interface: Input

Processes one element that arrived on this input of the MultipleInputStreamOperator. This method is guaranteed to not be called concurrently with other methods of the operator.

Specified by:

processElement in interface Input<IN>

Throws:

Exception

processWatermark

public void processWatermark(Watermark mark)
                      throws Exception

Description copied from interface: Input

Processes a Watermark that arrived on the first input of this two-input operator. This method is guaranteed to not be called concurrently with other methods of the operator.

Specified by:

processWatermark in interface Input<IN>

Throws:

Exception

See Also:

Watermark

processLatencyMarker

public void processLatencyMarker(LatencyMarker latencyMarker)
                          throws Exception

Description copied from interface: Input

Processes a LatencyMarker that arrived on the first input of this two-input operator. This method is guaranteed to not be called concurrently with other methods of the operator.

Specified by:

processLatencyMarker in interface Input<IN>

Throws:

Exception

See Also:

LatencyMarker

processWatermarkStatus

public void processWatermarkStatus(WatermarkStatus watermarkStatus)
                            throws Exception

Description copied from interface: Input

Processes a WatermarkStatus that arrived on this input of the MultipleInputStreamOperator. This method is guaranteed to not be called concurrently with other methods of the operator.

Specified by:

processWatermarkStatus in interface Input<IN>

Throws:

Exception

See Also:

WatermarkStatus

open

public void open()
          throws Exception

Description copied from interface: StreamOperator

This method is called immediately before any elements are processed, it should contain the operator's initialization logic.

Specified by:

open in interface StreamOperator<TaggedOperatorSubtaskState>

Throws:

Exception - An exception in this method causes the operator to fail.

finish

public void finish()
            throws Exception

Description copied from interface: StreamOperator

This method is called at the end of data processing.

The method is expected to flush all remaining buffered data. Exceptions during this flushing of buffered data should be propagated, in order to cause the operation to be recognized as failed, because the last data items are not processed properly.

After this method is called, no more records can be produced for the downstream operators.

WARNING: It is not safe to use this method to commit any transactions or other side effects! You can use this method to flush any buffered data that can later on be committed e.g. in a CheckpointListener.notifyCheckpointComplete(long).

NOTE:This method does not need to close any resources. You should release external resources in the StreamOperator.close() method.

Specified by:

finish in interface StreamOperator<TaggedOperatorSubtaskState>

Throws:

Exception - An exception in this method causes the operator to fail.

close

public void close()
           throws Exception

Description copied from interface: StreamOperator

This method is called at the very end of the operator's life, both in the case of a successful completion of the operation, and in the case of a failure and canceling.

This method is expected to make a thorough effort to release all resources that the operator has acquired.

NOTE:It can not emit any records! If you need to emit records at the end of processing, do so in the StreamOperator.finish() method.

Specified by:

close in interface StreamOperator<TaggedOperatorSubtaskState>

Throws:

Exception

prepareSnapshotPreBarrier

public void prepareSnapshotPreBarrier(long checkpointId)
                               throws Exception

Description copied from interface: StreamOperator

This method is called when the operator should do a snapshot, before it emits its own checkpoint barrier.

This method is intended not for any actual state persistence, but only for emitting some data before emitting the checkpoint barrier. Operators that maintain some small transient state that is inefficient to checkpoint (especially when it would need to be checkpointed in a re-scalable way) but can simply be sent downstream before the checkpoint. An example are opportunistic pre-aggregation operators, which have small the pre-aggregation state that is frequently flushed downstream.

Important: This method should not be used for any actual state snapshot logic, because it will inherently be within the synchronous part of the operator's checkpoint. If heavy work is done within this method, it will affect latency and downstream checkpoint alignments.

Specified by:

prepareSnapshotPreBarrier in interface StreamOperator<TaggedOperatorSubtaskState>

Parameters:

checkpointId - The ID of the checkpoint.

Throws:

Exception - Throwing an exception here causes the operator to fail and go into recovery.

snapshotState

public OperatorSnapshotFutures snapshotState(long checkpointId,
                                             long timestamp,
                                             CheckpointOptions checkpointOptions,
                                             CheckpointStreamFactory storageLocation)
                                      throws Exception

Description copied from interface: StreamOperator

Called to draw a state snapshot from the operator.

Specified by:

snapshotState in interface StreamOperator<TaggedOperatorSubtaskState>

Returns:

a runnable future to the state handle that points to the snapshotted state. For synchronous implementations, the runnable might already be finished.

Throws:

Exception - exception that happened during snapshotting.

initializeState

public void initializeState(StreamTaskStateInitializer streamTaskStateManager)
                     throws Exception

Description copied from interface: StreamOperator

Provides a context to initialize all state in the operator.

Specified by:

initializeState in interface StreamOperator<TaggedOperatorSubtaskState>

Throws:

Exception

setKeyContextElement1

public void setKeyContextElement1(StreamRecord<?> record)
                           throws Exception

Specified by:

setKeyContextElement1 in interface StreamOperator<TaggedOperatorSubtaskState>

Throws:

Exception

setKeyContextElement2

public void setKeyContextElement2(StreamRecord<?> record)
                           throws Exception

Specified by:

setKeyContextElement2 in interface StreamOperator<TaggedOperatorSubtaskState>

Throws:

Exception

getChainingStrategy

public ChainingStrategy getChainingStrategy()

Specified by:

getChainingStrategy in interface SetupableStreamOperator<TaggedOperatorSubtaskState>

setChainingStrategy

public void setChainingStrategy(ChainingStrategy strategy)

Specified by:

setChainingStrategy in interface SetupableStreamOperator<TaggedOperatorSubtaskState>

getMetricGroup

public OperatorMetricGroup getMetricGroup()

Specified by:

getMetricGroup in interface StreamOperator<TaggedOperatorSubtaskState>

getOperatorID

public OperatorID getOperatorID()

Specified by:

getOperatorID in interface StreamOperator<TaggedOperatorSubtaskState>

notifyCheckpointComplete

public void notifyCheckpointComplete(long checkpointId)
                              throws Exception

Description copied from interface: CheckpointListener

Notifies the listener that the checkpoint with the given checkpointId completed and was committed.

These notifications are "best effort", meaning they can sometimes be skipped. To behave properly, implementers need to follow the "Checkpoint Subsuming Contract". Please see the class-level JavaDocs for details.

Please note that checkpoints may generally overlap, so you cannot assume that the notifyCheckpointComplete() call is always for the latest prior checkpoint (or snapshot) that was taken on the function/operator implementing this interface. It might be for a checkpoint that was triggered earlier. Implementing the "Checkpoint Subsuming Contract" (see above) properly handles this situation correctly as well.

Please note that throwing exceptions from this method will not cause the completed checkpoint to be revoked. Throwing exceptions will typically cause task/job failure and trigger recovery.

Specified by:

notifyCheckpointComplete in interface CheckpointListener

Parameters:

checkpointId - The ID of the checkpoint that has been completed.

Throws:

Exception - This method can propagate exceptions, which leads to a failure/recovery for the task. Note that this will NOT lead to the checkpoint being revoked.

setCurrentKey

public void setCurrentKey(Object key)

Specified by:

setCurrentKey in interface KeyContext

getCurrentKey

public Object getCurrentKey()

Specified by:

getCurrentKey in interface KeyContext

endInput

public void endInput()
              throws Exception

Description copied from interface: BoundedOneInput

It is notified that no more data will arrive from the input.

WARNING: It is not safe to use this method to commit any transactions or other side effects! You can use this method to flush any buffered data that can later on be committed e.g. in a CheckpointListener.notifyCheckpointComplete(long).

NOTE: Given it is semantically very similar to the StreamOperator.finish() method. It might be dropped in favour of the other method at some point in time.

Specified by:

endInput in interface BoundedOneInput

Throws:

Exception