@Internal public class ExternalPythonKeyedCoProcessOperator<OUT> extends AbstractExternalTwoInputPythonFunctionOperator<Row,Row,OUT> implements Triggerable<Row,VoidNamespace>
ExternalPythonKeyedCoProcessOperator is responsible for launching beam runner which will
start a python harness to execute user defined python CoProcess function. It is also able to
handle the timer and state request from the python stateful user defined function.baos, baosWrapperpythonFunctionRunnerbundleFinishedCallback, config, elementCount, lastFinishBundleTime, maxBundleSize, systemEnvEnabledchainingStrategy, latencyStats, LOG, metrics, output, processingTimeService| Constructor and Description |
|---|
ExternalPythonKeyedCoProcessOperator(Configuration config,
DataStreamPythonFunctionInfo pythonFunctionInfo,
TypeInformation<Row> inputTypeInfo1,
TypeInformation<Row> inputTypeInfo2,
TypeInformation<OUT> outputTypeInfo) |
| Modifier and Type | Method and Description |
|---|---|
<T> AbstractExternalDataStreamPythonFunctionOperator<T> |
copy(DataStreamPythonFunctionInfo pythonFunctionInfo,
TypeInformation<T> outputTypeInfo)
Make a copy of the DataStreamPythonFunctionOperator with the given pythonFunctionInfo and
outputTypeInfo.
|
PythonFunctionRunner |
createPythonFunctionRunner()
Creates the
PythonFunctionRunner which is responsible for Python user-defined
function execution. |
Object |
getCurrentKey() |
void |
onEventTime(InternalTimer<Row,VoidNamespace> timer)
Invoked when an event-time timer fires.
|
void |
onProcessingTime(InternalTimer<Row,VoidNamespace> timer)
Invoked when a processing-time timer fires.
|
void |
open()
This method is called immediately before any elements are processed, it should contain the
operator's initialization logic, e.g.
|
void |
processElement1(StreamRecord<Row> element)
Processes one element that arrived on the first input of this two-input operator.
|
void |
processElement2(StreamRecord<Row> element)
Processes one element that arrived on the second input of this two-input operator.
|
void |
setCurrentKey(Object key)
As the beam state gRPC service will access the KeyedStateBackend in parallel with this
operator, we must override this method to prevent changing the current key of the
KeyedStateBackend while the beam service is handling requests.
|
createInputCoderInfoDescriptor, createOutputCoderInfoDescriptor, emitResult, endInput, getLeftInputType, getRightInputType, processElementaddSideOutputTags, createSideOutputCoderDescriptors, getInternalParameters, getOutputTagById, getProducedType, getPythonEnv, getPythonFunctionInfo, getSideOutputTags, getSideOutputTypeSerializerById, setNumPartitionsclose, createPythonEnvironmentManager, emitResults, invokeFinishBundlecheckInvokeFinishBundleByCount, finish, getConfiguration, getFlinkMetricContainer, isBundleFinished, prepareSnapshotPreBarrier, processWatermarkgetChainingStrategy, getContainingTask, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setChainingStrategy, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState, snapshotStateclone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitprocessLatencyMarker1, processLatencyMarker2, processWatermark1, processWatermark2, processWatermarkStatus1, processWatermarkStatus2close, finish, getMetricGroup, getOperatorID, initializeState, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotStatenotifyCheckpointAborted, notifyCheckpointCompletehasKeyContextpublic ExternalPythonKeyedCoProcessOperator(Configuration config, DataStreamPythonFunctionInfo pythonFunctionInfo, TypeInformation<Row> inputTypeInfo1, TypeInformation<Row> inputTypeInfo2, TypeInformation<OUT> outputTypeInfo)
public void open()
throws Exception
AbstractStreamOperatorThe default implementation does nothing.
open in interface StreamOperator<OUT>open in class AbstractExternalTwoInputPythonFunctionOperator<Row,Row,OUT>Exception - An exception in this method causes the operator to fail.public PythonFunctionRunner createPythonFunctionRunner() throws Exception
AbstractExternalPythonFunctionOperatorPythonFunctionRunner which is responsible for Python user-defined
function execution.createPythonFunctionRunner in class AbstractExternalPythonFunctionOperator<OUT>Exceptionpublic void processElement1(StreamRecord<Row> element) throws Exception
TwoInputStreamOperatorprocessElement1 in interface TwoInputStreamOperator<Row,Row,OUT>Exceptionpublic void processElement2(StreamRecord<Row> element) throws Exception
TwoInputStreamOperatorprocessElement2 in interface TwoInputStreamOperator<Row,Row,OUT>Exceptionpublic void onEventTime(InternalTimer<Row,VoidNamespace> timer) throws Exception
TriggerableonEventTime in interface Triggerable<Row,VoidNamespace>Exceptionpublic void onProcessingTime(InternalTimer<Row,VoidNamespace> timer) throws Exception
TriggerableonProcessingTime in interface Triggerable<Row,VoidNamespace>Exceptionpublic void setCurrentKey(Object key)
setCurrentKey in interface KeyContextsetCurrentKey in class AbstractPythonFunctionOperator<OUT>public Object getCurrentKey()
getCurrentKey in interface KeyContextgetCurrentKey in class AbstractStreamOperator<OUT>public <T> AbstractExternalDataStreamPythonFunctionOperator<T> copy(DataStreamPythonFunctionInfo pythonFunctionInfo, TypeInformation<T> outputTypeInfo)
DataStreamPythonFunctionOperatorcopy in interface DataStreamPythonFunctionOperator<OUT>Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.