@Internal public class PartitionCommitter extends AbstractStreamOperator<Void> implements OneInputStreamOperator<PartitionCommitInfo,Void>
NOTE: It processes records after the checkpoint completes successfully. Receive records from
upstream CheckpointListener.notifyCheckpointComplete(long).
Processing steps: 1.Partitions are sent from upstream. Add partition to trigger. 2.TaskTracker say it have already received partition data from all tasks in a checkpoint.
3.Extracting committable partitions from PartitionCommitTrigger. 4.Using PartitionCommitPolicy chain to commit partitions.
chainingStrategy, config, lastRecordAttributes1, lastRecordAttributes2, latencyStats, LOG, metrics, output, processingTimeService| Constructor and Description |
|---|
PartitionCommitter(Path locationPath,
ObjectIdentifier tableIdentifier,
List<String> partitionKeys,
TableMetaStoreFactory metaStoreFactory,
FileSystemFactory fsFactory,
Configuration conf) |
| Modifier and Type | Method and Description |
|---|---|
void |
initializeState(StateInitializationContext context)
Stream operators with state which can be restored need to override this hook method.
|
void |
processElement(StreamRecord<PartitionCommitInfo> element)
Processes one element that arrived on this input of the
MultipleInputStreamOperator. |
void |
processWatermark(Watermark mark)
Processes a
Watermark that arrived on the first input of this two-input operator. |
void |
snapshotState(StateSnapshotContext context)
Stream operators with state, which want to participate in a snapshot need to override this
hook method.
|
close, finish, getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, open, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotStateclone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitsetKeyContextElementclose, finish, getMetricGroup, getOperatorID, initializeState, open, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotStatenotifyCheckpointAborted, notifyCheckpointCompletegetCurrentKey, setCurrentKeyprocessLatencyMarker, processRecordAttributes, processWatermarkStatushasKeyContextpublic PartitionCommitter(Path locationPath, ObjectIdentifier tableIdentifier, List<String> partitionKeys, TableMetaStoreFactory metaStoreFactory, FileSystemFactory fsFactory, Configuration conf)
public void initializeState(StateInitializationContext context) throws Exception
AbstractStreamOperatorinitializeState in interface StreamOperatorStateHandler.CheckpointedStreamOperatorinitializeState in class AbstractStreamOperator<Void>context - context that allows to register different states.Exceptionpublic void processElement(StreamRecord<PartitionCommitInfo> element) throws Exception
InputMultipleInputStreamOperator.
This method is guaranteed to not be called concurrently with other methods of the operator.processElement in interface Input<PartitionCommitInfo>Exceptionpublic void processWatermark(Watermark mark) throws Exception
InputWatermark that arrived on the first input of this two-input operator.
This method is guaranteed to not be called concurrently with other methods of the operator.processWatermark in interface Input<PartitionCommitInfo>processWatermark in class AbstractStreamOperator<Void>ExceptionWatermarkpublic void snapshotState(StateSnapshotContext context) throws Exception
AbstractStreamOperatorsnapshotState in interface StreamOperatorStateHandler.CheckpointedStreamOperatorsnapshotState in class AbstractStreamOperator<Void>context - context that provides information and means required for taking a snapshotExceptionCopyright © 2014–2024 The Apache Software Foundation. All rights reserved.