@Internal public class CompactCoordinator extends AbstractStreamOperator<CompactMessages.CoordinatorOutput> implements OneInputStreamOperator<CompactMessages.CoordinatorInput,CompactMessages.CoordinatorOutput>
CompactMessages.CompactionUnit
and CompactMessages.EndCompaction
must be sent to the downstream in an orderly
manner, while CompactMessages.EndCompaction
is broadcast emitting, so unit and endCompaction use the
broadcast emitting mechanism together. Since unit is broadcast, we want it to be processed by a
single task, so we carry the ID in the unit and let the downstream task select its own unit.
NOTE: The coordination is a stable algorithm, which can ensure that the downstream can perform compaction at any time without worrying about fail over.
STATE: This operator stores input files in state, after the checkpoint completes successfully, input files are taken out from the state for coordination.
chainingStrategy, config, lastRecordAttributes1, lastRecordAttributes2, latencyStats, metrics, output, processingTimeService, stateHandler, stateKeySelector1, stateKeySelector2, timeServiceManager
Constructor and Description |
---|
CompactCoordinator(SupplierWithException<FileSystem,IOException> fsFactory,
long targetFileSize) |
Modifier and Type | Method and Description |
---|---|
void |
initializeState(StateInitializationContext context)
Stream operators with state which can be restored need to override this hook method.
|
void |
processElement(StreamRecord<CompactMessages.CoordinatorInput> element)
Processes one element that arrived on this input of the
MultipleInputStreamOperator . |
void |
snapshotState(StateSnapshotContext context)
Stream operators with state, which want to participate in a snapshot need to override this
hook method.
|
close, finish, getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, open, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setMailboxExecutor, setProcessingTimeService, setup, snapshotState, useSplittableTimers
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
setKeyContextElement
close, finish, getMetricGroup, getOperatorAttributes, getOperatorID, initializeState, open, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState
notifyCheckpointAborted, notifyCheckpointComplete
getCurrentKey, setCurrentKey
processLatencyMarker, processRecordAttributes, processWatermark, processWatermarkStatus
hasKeyContext
public CompactCoordinator(SupplierWithException<FileSystem,IOException> fsFactory, long targetFileSize)
public void initializeState(StateInitializationContext context) throws Exception
AbstractStreamOperator
initializeState
in interface StreamOperatorStateHandler.CheckpointedStreamOperator
initializeState
in class AbstractStreamOperator<CompactMessages.CoordinatorOutput>
context
- context that allows to register different states.Exception
public void processElement(StreamRecord<CompactMessages.CoordinatorInput> element) throws Exception
Input
MultipleInputStreamOperator
.
This method is guaranteed to not be called concurrently with other methods of the operator.processElement
in interface Input<CompactMessages.CoordinatorInput>
Exception
public void snapshotState(StateSnapshotContext context) throws Exception
AbstractStreamOperator
snapshotState
in interface StreamOperatorStateHandler.CheckpointedStreamOperator
snapshotState
in class AbstractStreamOperator<CompactMessages.CoordinatorOutput>
context
- context that provides information and means required for taking a snapshotException
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.