@Internal public class ContinuousFileReaderOperator<OUT> extends AbstractStreamOperator<OUT> implements OneInputStreamOperator<TimestampedFileInputSplit,OUT>, OutputTypeConfigurable<OUT>
splits
received from the preceding
ContinuousFileMonitoringFunction
. Contrary to the ContinuousFileMonitoringFunction
which has a parallelism of 1, this operator can have DOP > 1.
As soon as a split descriptor is received, it is put in a queue, and have another thread read the actual data of the split. This architecture allows the separation of the reading thread from the one emitting the checkpoint barriers, thus removing any potential back-pressure.
AbstractStreamOperator.CountingOutput<OUT>
chainingStrategy, config, latencyStats, metrics, output, timeServiceManager
Constructor and Description |
---|
ContinuousFileReaderOperator(FileInputFormat<OUT> format) |
Modifier and Type | Method and Description |
---|---|
void |
close()
This method is called after all records have been added to the operators via the methods
OneInputStreamOperator.processElement(StreamRecord) , or
TwoInputStreamOperator.processElement1(StreamRecord) and
TwoInputStreamOperator.processElement2(StreamRecord) . |
void |
dispose()
This method is called at the very end of the operator's life, both in the case of a successful
completion of the operation, and in the case of a failure and canceling.
|
void |
initializeState(StateInitializationContext context)
Stream operators with state which can be restored need to override this hook method.
|
void |
open()
This method is called immediately before any elements are processed, it should contain the
operator's initialization logic, e.g.
|
void |
processElement(StreamRecord<TimestampedFileInputSplit> element)
Processes one element that arrived at this operator.
|
void |
processWatermark(Watermark mark)
Processes a
Watermark . |
void |
setOutputType(TypeInformation<OUT> outTypeInfo,
ExecutionConfig executionConfig)
Is called by the
org.apache.flink.streaming.api.graph.StreamGraph#addOperator(Integer, String, StreamOperator, TypeInformation, TypeInformation, String)
method when the StreamGraph is generated. |
void |
snapshotState(StateSnapshotContext context)
Stream operators with state, which want to participate in a snapshot need to override this hook method.
|
getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getUserCodeClassloader, initializeState, notifyCheckpointComplete, numEventTimeTimers, numProcessingTimeTimers, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark1, processWatermark2, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setup, snapshotState
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
processLatencyMarker
getChainingStrategy, getMetricGroup, getOperatorID, initializeState, prepareSnapshotPreBarrier, setChainingStrategy, setKeyContextElement1, setKeyContextElement2, setup, snapshotState
notifyCheckpointComplete
getCurrentKey, setCurrentKey
public ContinuousFileReaderOperator(FileInputFormat<OUT> format)
public void setOutputType(TypeInformation<OUT> outTypeInfo, ExecutionConfig executionConfig)
OutputTypeConfigurable
org.apache.flink.streaming.api.graph.StreamGraph#addOperator(Integer, String, StreamOperator, TypeInformation, TypeInformation, String)
method when the StreamGraph
is generated. The
method is called with the output TypeInformation
which is also used for the
StreamTask
output serializer.setOutputType
in interface OutputTypeConfigurable<OUT>
outTypeInfo
- Output type information of the StreamTask
executionConfig
- Execution configurationpublic void initializeState(StateInitializationContext context) throws Exception
AbstractStreamOperator
initializeState
in class AbstractStreamOperator<OUT>
context
- context that allows to register different states.Exception
public void open() throws Exception
AbstractStreamOperator
The default implementation does nothing.
open
in interface StreamOperator<OUT>
open
in class AbstractStreamOperator<OUT>
Exception
- An exception in this method causes the operator to fail.public void processElement(StreamRecord<TimestampedFileInputSplit> element) throws Exception
OneInputStreamOperator
processElement
in interface OneInputStreamOperator<TimestampedFileInputSplit,OUT>
Exception
public void processWatermark(Watermark mark) throws Exception
OneInputStreamOperator
Watermark
.
This method is guaranteed to not be called concurrently with other methods of the operator.processWatermark
in interface OneInputStreamOperator<TimestampedFileInputSplit,OUT>
processWatermark
in class AbstractStreamOperator<OUT>
Exception
Watermark
public void dispose() throws Exception
AbstractStreamOperator
This method is expected to make a thorough effort to release all resources that the operator has acquired.
dispose
in interface StreamOperator<OUT>
dispose
in interface Disposable
dispose
in class AbstractStreamOperator<OUT>
Exception
- if something goes wrong during disposal.public void close() throws Exception
AbstractStreamOperator
OneInputStreamOperator.processElement(StreamRecord)
, or
TwoInputStreamOperator.processElement1(StreamRecord)
and
TwoInputStreamOperator.processElement2(StreamRecord)
.
The method is expected to flush all remaining buffered data. Exceptions during this flushing of buffered should be propagated, in order to cause the operation to be recognized asa failed, because the last data items are not processed properly.
close
in interface StreamOperator<OUT>
close
in class AbstractStreamOperator<OUT>
Exception
- An exception in this method causes the operator to fail.public void snapshotState(StateSnapshotContext context) throws Exception
AbstractStreamOperator
snapshotState
in class AbstractStreamOperator<OUT>
context
- context that provides information and means required for taking a snapshotException
Copyright © 2014–2020 The Apache Software Foundation. All rights reserved.