public class StreamingFileCommitter extends AbstractStreamOperator<Void> implements OneInputStreamOperator<StreamingFileCommitter.CommitMessage,Void>
StreamingFileWriter
. This is the single (non-parallel) task. It collects
all the partition information sent from upstream, and triggers the partition submission decision
when it judges to collect the partitions from all tasks of a checkpoint.
Processing steps: 1.Partitions are sent from upstream. Add partition to trigger. 2.TaskTracker
say it have already received partition data from all tasks in a checkpoint.
3.Extracting committable partitions from PartitionCommitTrigger
. 4.Using PartitionCommitPolicy
chain to commit partitions.
Modifier and Type | Class and Description |
---|---|
static class |
StreamingFileCommitter.CommitMessage
The message sent upstream.
|
chainingStrategy, config, latencyStats, LOG, metrics, output, processingTimeService
Constructor and Description |
---|
StreamingFileCommitter(Path locationPath,
ObjectIdentifier tableIdentifier,
List<String> partitionKeys,
TableMetaStoreFactory metaStoreFactory,
FileSystemFactory fsFactory,
Configuration conf) |
Modifier and Type | Method and Description |
---|---|
void |
initializeState(StateInitializationContext context)
Stream operators with state which can be restored need to override this hook method.
|
void |
processElement(StreamRecord<StreamingFileCommitter.CommitMessage> element)
Processes one element that arrived at this operator.
|
void |
processWatermark(Watermark mark)
Processes a
Watermark . |
void |
snapshotState(StateSnapshotContext context)
Stream operators with state, which want to participate in a snapshot need to override this
hook method.
|
close, dispose, getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, numEventTimeTimers, numProcessingTimeTimers, open, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark1, processWatermark2, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
processLatencyMarker
close, dispose, getMetricGroup, getOperatorID, initializeState, open, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState
notifyCheckpointAborted, notifyCheckpointComplete
getCurrentKey, setCurrentKey
public StreamingFileCommitter(Path locationPath, ObjectIdentifier tableIdentifier, List<String> partitionKeys, TableMetaStoreFactory metaStoreFactory, FileSystemFactory fsFactory, Configuration conf)
public void initializeState(StateInitializationContext context) throws Exception
AbstractStreamOperator
initializeState
in interface StreamOperatorStateHandler.CheckpointedStreamOperator
initializeState
in class AbstractStreamOperator<Void>
context
- context that allows to register different states.Exception
public void processElement(StreamRecord<StreamingFileCommitter.CommitMessage> element) throws Exception
OneInputStreamOperator
processElement
in interface OneInputStreamOperator<StreamingFileCommitter.CommitMessage,Void>
Exception
public void processWatermark(Watermark mark) throws Exception
OneInputStreamOperator
Watermark
. This method is guaranteed to not be called concurrently with
other methods of the operator.processWatermark
in interface OneInputStreamOperator<StreamingFileCommitter.CommitMessage,Void>
processWatermark
in class AbstractStreamOperator<Void>
Exception
Watermark
public void snapshotState(StateSnapshotContext context) throws Exception
AbstractStreamOperator
snapshotState
in interface StreamOperatorStateHandler.CheckpointedStreamOperator
snapshotState
in class AbstractStreamOperator<Void>
context
- context that provides information and means required for taking a snapshotException
Copyright © 2014–2021 The Apache Software Foundation. All rights reserved.