@Internal public class PartitionCommitter extends AbstractStreamOperator<Void> implements OneInputStreamOperator<PartitionCommitInfo,Void>
NOTE: It processes records after the checkpoint completes successfully. Receive records from
upstream CheckpointListener.notifyCheckpointComplete(long)
.
Processing steps: 1.Partitions are sent from upstream. Add partition to trigger. 2.TaskTracker
say it have already received partition data from all tasks in a checkpoint.
3.Extracting committable partitions from PartitionCommitTrigger
. 4.Using PartitionCommitPolicy
chain to commit partitions.
chainingStrategy, config, lastRecordAttributes1, lastRecordAttributes2, latencyStats, LOG, metrics, output, processingTimeService, stateHandler, stateKeySelector1, stateKeySelector2, timeServiceManager
Constructor and Description |
---|
PartitionCommitter(Path locationPath,
ObjectIdentifier tableIdentifier,
List<String> partitionKeys,
TableMetaStoreFactory metaStoreFactory,
FileSystemFactory fsFactory,
Configuration conf) |
Modifier and Type | Method and Description |
---|---|
void |
initializeState(StateInitializationContext context)
Stream operators with state which can be restored need to override this hook method.
|
void |
processElement(StreamRecord<PartitionCommitInfo> element)
Processes one element that arrived on this input of the
MultipleInputStreamOperator . |
void |
processWatermark(Watermark mark)
Processes a
Watermark that arrived on the first input of this two-input operator. |
void |
snapshotState(StateSnapshotContext context)
Stream operators with state, which want to participate in a snapshot need to override this
hook method.
|
close, finish, getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, open, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setMailboxExecutor, setProcessingTimeService, setup, snapshotState, useSplittableTimers
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
setKeyContextElement
close, finish, getMetricGroup, getOperatorAttributes, getOperatorID, initializeState, open, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState
notifyCheckpointAborted, notifyCheckpointComplete
getCurrentKey, setCurrentKey
processLatencyMarker, processRecordAttributes, processWatermarkStatus
hasKeyContext
public PartitionCommitter(Path locationPath, ObjectIdentifier tableIdentifier, List<String> partitionKeys, TableMetaStoreFactory metaStoreFactory, FileSystemFactory fsFactory, Configuration conf)
public void initializeState(StateInitializationContext context) throws Exception
AbstractStreamOperator
initializeState
in interface StreamOperatorStateHandler.CheckpointedStreamOperator
initializeState
in class AbstractStreamOperator<Void>
context
- context that allows to register different states.Exception
public void processElement(StreamRecord<PartitionCommitInfo> element) throws Exception
Input
MultipleInputStreamOperator
.
This method is guaranteed to not be called concurrently with other methods of the operator.processElement
in interface Input<PartitionCommitInfo>
Exception
public void processWatermark(Watermark mark) throws Exception
Input
Watermark
that arrived on the first input of this two-input operator.
This method is guaranteed to not be called concurrently with other methods of the operator.processWatermark
in interface Input<PartitionCommitInfo>
processWatermark
in class AbstractStreamOperator<Void>
Exception
Watermark
public void snapshotState(StateSnapshotContext context) throws Exception
AbstractStreamOperator
snapshotState
in interface StreamOperatorStateHandler.CheckpointedStreamOperator
snapshotState
in class AbstractStreamOperator<Void>
context
- context that provides information and means required for taking a snapshotException
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.