IN
- Type of the elements emitted by this sinkpublic abstract class GenericWriteAheadSink<IN> extends AbstractStreamOperator<IN> implements OneInputStreamOperator<IN,IN>
AbstractStateBackend
, and only committed if a
checkpoint is completed.Modifier and Type | Class and Description |
---|---|
static class |
GenericWriteAheadSink.ExactlyOnceState
This state is used to keep a list of all StateHandles (essentially references to past OperatorStates) that were
used since the last completed checkpoint.
|
AbstractStreamOperator.CountingOutput
Modifier and Type | Field and Description |
---|---|
protected static org.slf4j.Logger |
LOG |
protected TypeSerializer<IN> |
serializer |
chainingStrategy, metrics, output
Constructor and Description |
---|
GenericWriteAheadSink(CheckpointCommitter committer,
TypeSerializer<IN> serializer,
String jobID) |
Modifier and Type | Method and Description |
---|---|
void |
close()
This method is called after all records have been added to the operators via the methods
OneInputStreamOperator.processElement(StreamRecord) , or
TwoInputStreamOperator.processElement1(StreamRecord) and
TwoInputStreamOperator.processElement2(StreamRecord) . |
void |
notifyOfCompletedCheckpoint(long checkpointId)
Called when the checkpoint with the given ID is completed and acknowledged on the JobManager.
|
void |
open()
This method is called immediately before any elements are processed, it should contain the
operator's initialization logic.
|
void |
processElement(StreamRecord<IN> element)
Processes one element that arrived at this operator.
|
void |
processWatermark(Watermark mark)
Processes a
Watermark . |
void |
restoreState(StreamTaskState state)
Restores the operator state, if this operator's execution is recovering from a checkpoint.
|
protected abstract boolean |
sendValues(Iterable<IN> value,
long timestamp)
Write the given element into the backend.
|
StreamTaskState |
snapshotOperatorState(long checkpointId,
long timestamp)
Called to draw a state snapshot from the operator.
|
dispose, getChainingStrategy, getContainingTask, getCurrentProcessingTime, getExecutionConfig, getMetricGroup, getOperatorConfig, getOperatorName, getPartitionedState, getPartitionedState, getRuntimeContext, getStateBackend, getUserCodeClassloader, registerTimer, setChainingStrategy, setKeyContext, setKeyContextElement1, setKeyContextElement2, setup
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
dispose, getChainingStrategy, getMetricGroup, setChainingStrategy, setKeyContextElement1, setKeyContextElement2, setup
protected static final org.slf4j.Logger LOG
protected final TypeSerializer<IN> serializer
public GenericWriteAheadSink(CheckpointCommitter committer, TypeSerializer<IN> serializer, String jobID) throws Exception
Exception
public void open() throws Exception
AbstractStreamOperator
The default implementation does nothing.
open
in interface StreamOperator<IN>
open
in class AbstractStreamOperator<IN>
Exception
- An exception in this method causes the operator to fail.public void close() throws Exception
AbstractStreamOperator
OneInputStreamOperator.processElement(StreamRecord)
, or
TwoInputStreamOperator.processElement1(StreamRecord)
and
TwoInputStreamOperator.processElement2(StreamRecord)
.
The method is expected to flush all remaining buffered data. Exceptions during this flushing of buffered should be propagated, in order to cause the operation to be recognized asa failed, because the last data items are not processed properly.
close
in interface StreamOperator<IN>
close
in class AbstractStreamOperator<IN>
Exception
- An exception in this method causes the operator to fail.public StreamTaskState snapshotOperatorState(long checkpointId, long timestamp) throws Exception
StreamOperator
snapshotOperatorState
in interface StreamOperator<IN>
snapshotOperatorState
in class AbstractStreamOperator<IN>
checkpointId
- The ID of the checkpoint.timestamp
- The timestamp of the checkpoint.Exception
- Forwards exceptions that occur while drawing snapshots from the operator
and the key/value state.public void restoreState(StreamTaskState state) throws Exception
StreamOperator
This method is called after StreamOperator.setup(StreamTask, StreamConfig, Output)
and before StreamOperator.open()
.
restoreState
in interface StreamOperator<IN>
restoreState
in class AbstractStreamOperator<IN>
state
- The state of operator that was snapshotted as part of checkpoint
from which the execution is restored.Exception
- Exceptions during state restore should be forwarded, so that the system can
properly react to failed state restore and fail the execution attempt.public void notifyOfCompletedCheckpoint(long checkpointId) throws Exception
StreamOperator
notifyOfCompletedCheckpoint
in interface StreamOperator<IN>
notifyOfCompletedCheckpoint
in class AbstractStreamOperator<IN>
checkpointId
- The ID of the checkpoint that has been completed.Exception
- Exceptions during checkpoint acknowledgement may be forwarded and will cause
the program to fail and enter recovery.protected abstract boolean sendValues(Iterable<IN> value, long timestamp) throws Exception
value
- value to be writtenException
public void processElement(StreamRecord<IN> element) throws Exception
OneInputStreamOperator
processElement
in interface OneInputStreamOperator<IN,IN>
Exception
public void processWatermark(Watermark mark) throws Exception
OneInputStreamOperator
Watermark
.
This method is guaranteed to not be called concurrently with other methods of the operator.processWatermark
in interface OneInputStreamOperator<IN,IN>
Exception
Watermark
Copyright © 2014–2017 The Apache Software Foundation. All rights reserved.