public class TemporalRowTimeJoinOperator extends BaseTwoInputStreamOperatorWithStateRetention
For Event-time temporal join, its probe side is a regular table, its build side is a versioned table, the version of versioned table can extract from the build side state. This operator works by keeping on the state collection of probe and build records to process on next watermark. The idea is that between watermarks we are collecting those elements and once we are sure that there will be no updates we emit the correct result and clean up the expired data in state.
Cleaning up the state drops all of the "old" values from the probe side, where "old" is defined as older then the current watermark. Build side is also cleaned up in the similar fashion, however we always keep at least one record - the latest one - even if it's past the last watermark.
One more trick is how the emitting results and cleaning up is triggered. It is achieved by registering timers for the keys. We could register a timer for every probe and build side element's event time (when watermark exceeds this timer, that's when we are emitting and/or cleaning up the state). However this would cause huge number of registered timers. For example with following evenTimes of probe records accumulated: {1, 2, 5, 8, 9}, if we had received Watermark(10), it would trigger 5 separate timers for the same key. To avoid that we always keep only one single registered timer for any given key, registered for the minimal value. Upon triggering it, we process all records with event times older then or equal to currentWatermark.
stateCleaningEnabledchainingStrategy, config, latencyStats, LOG, metrics, output, processingTimeService| Constructor and Description |
|---|
TemporalRowTimeJoinOperator(InternalTypeInfo<RowData> leftType,
InternalTypeInfo<RowData> rightType,
GeneratedJoinCondition generatedJoinCondition,
int leftTimeAttribute,
int rightTimeAttribute,
long minRetentionTime,
long maxRetentionTime,
boolean isLeftOuterJoin) |
| Modifier and Type | Method and Description |
|---|---|
void |
cleanupState(long time)
The method to be called when a cleanup timer fires.
|
void |
close()
This method is called at the very end of the operator's life, both in the case of a
successful completion of the operation, and in the case of a failure and canceling.
|
void |
onEventTime(InternalTimer<Object,VoidNamespace> timer)
Invoked when an event-time timer fires.
|
void |
open()
This method is called immediately before any elements are processed, it should contain the
operator's initialization logic, e.g.
|
void |
processElement1(StreamRecord<RowData> element)
Processes one element that arrived on the first input of this two-input operator.
|
void |
processElement2(StreamRecord<RowData> element)
Processes one element that arrived on the second input of this two-input operator.
|
cleanupLastTimer, onProcessingTime, registerProcessingCleanupTimerfinish, getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, initializeState, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, registerCounterOnOutput, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState, snapshotStateclone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitprocessLatencyMarker1, processLatencyMarker2, processWatermark1, processWatermark2, processWatermarkStatus1, processWatermarkStatus2finish, getMetricGroup, getOperatorID, initializeState, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotStatenotifyCheckpointAborted, notifyCheckpointCompletegetCurrentKey, setCurrentKeypublic TemporalRowTimeJoinOperator(InternalTypeInfo<RowData> leftType, InternalTypeInfo<RowData> rightType, GeneratedJoinCondition generatedJoinCondition, int leftTimeAttribute, int rightTimeAttribute, long minRetentionTime, long maxRetentionTime, boolean isLeftOuterJoin)
public void open()
throws Exception
AbstractStreamOperatorThe default implementation does nothing.
open in interface StreamOperator<RowData>open in class BaseTwoInputStreamOperatorWithStateRetentionException - An exception in this method causes the operator to fail.public void processElement1(StreamRecord<RowData> element) throws Exception
TwoInputStreamOperatorExceptionpublic void processElement2(StreamRecord<RowData> element) throws Exception
TwoInputStreamOperatorExceptionpublic void onEventTime(InternalTimer<Object,VoidNamespace> timer) throws Exception
TriggerableExceptionpublic void close()
throws Exception
StreamOperatorThis method is expected to make a thorough effort to release all resources that the operator has acquired.
NOTE:It can not emit any records! If you need to emit records at the end of
processing, do so in the StreamOperator.finish() method.
close in interface StreamOperator<RowData>close in class AbstractStreamOperator<RowData>Exceptionpublic void cleanupState(long time)
cleanupState in class BaseTwoInputStreamOperatorWithStateRetentiontime - The timestamp of the fired timer.Copyright © 2014–2023 The Apache Software Foundation. All rights reserved.