@Internal public class ExternalPythonProcessOperator<IN,OUT> extends AbstractExternalOneInputPythonFunctionOperator<IN,OUT>
ExternalPythonProcessOperator
is responsible for launching beam runner which will start a
python harness to execute user defined python ProcessFunction.baos, baosWrapper
pythonFunctionRunner
bundleFinishedCallback, config, elementCount, lastFinishBundleTime, maxBundleSize, systemEnvEnabled
chainingStrategy, lastRecordAttributes1, lastRecordAttributes2, latencyStats, LOG, metrics, output, processingTimeService, stateHandler, stateKeySelector1, stateKeySelector2, timeServiceManager
Constructor and Description |
---|
ExternalPythonProcessOperator(Configuration config,
DataStreamPythonFunctionInfo pythonFunctionInfo,
TypeInformation<IN> inputTypeInfo,
TypeInformation<OUT> outputTypeInfo) |
Modifier and Type | Method and Description |
---|---|
<T> AbstractExternalDataStreamPythonFunctionOperator<T> |
copy(DataStreamPythonFunctionInfo pythonFunctionInfo,
TypeInformation<T> outputTypeInfo)
Make a copy of the DataStreamPythonFunctionOperator with the given pythonFunctionInfo and
outputTypeInfo.
|
PythonFunctionRunner |
createPythonFunctionRunner()
Creates the
PythonFunctionRunner which is responsible for Python user-defined
function execution. |
void |
open()
This method is called immediately before any elements are processed, it should contain the
operator's initialization logic, e.g. state initialization.
|
void |
processElement(StreamRecord<IN> element)
Processes one element that arrived on this input of the
MultipleInputStreamOperator . |
void |
processWatermark(Watermark mark)
Processes a
Watermark that arrived on the first input of this two-input operator. |
createInputCoderInfoDescriptor, createOutputCoderInfoDescriptor, emitResult, endInput, getInputTypeInfo, processElement
addSideOutputTags, createSideOutputCoderDescriptors, getInternalParameters, getOutputTagById, getProducedType, getPythonEnv, getPythonFunctionInfo, getSideOutputTags, getSideOutputTypeSerializerById, setNumPartitions
close, createPythonEnvironmentManager, emitResults, invokeFinishBundle
checkInvokeFinishBundleByCount, finish, getConfiguration, getFlinkMetricContainer, isBundleFinished, prepareSnapshotPreBarrier, setCurrentKey
getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setChainingStrategy, setKeyContextElement1, setKeyContextElement2, setMailboxExecutor, setProcessingTimeService, setup, snapshotState, snapshotState, useSplittableTimers
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
setKeyContextElement
close, finish, getMetricGroup, getOperatorAttributes, getOperatorID, initializeState, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState
notifyCheckpointAborted, notifyCheckpointComplete
getCurrentKey, setCurrentKey
processLatencyMarker, processRecordAttributes, processWatermarkStatus
hasKeyContext
public ExternalPythonProcessOperator(Configuration config, DataStreamPythonFunctionInfo pythonFunctionInfo, TypeInformation<IN> inputTypeInfo, TypeInformation<OUT> outputTypeInfo)
public void open() throws Exception
AbstractStreamOperator
The default implementation does nothing.
open
in interface StreamOperator<OUT>
open
in class AbstractExternalOneInputPythonFunctionOperator<IN,OUT>
Exception
- An exception in this method causes the operator to fail.public PythonFunctionRunner createPythonFunctionRunner() throws Exception
AbstractExternalPythonFunctionOperator
PythonFunctionRunner
which is responsible for Python user-defined
function execution.createPythonFunctionRunner
in class AbstractExternalPythonFunctionOperator<OUT>
Exception
public void processElement(StreamRecord<IN> element) throws Exception
Input
MultipleInputStreamOperator
.
This method is guaranteed to not be called concurrently with other methods of the operator.Exception
public void processWatermark(Watermark mark) throws Exception
Input
Watermark
that arrived on the first input of this two-input operator.
This method is guaranteed to not be called concurrently with other methods of the operator.processWatermark
in interface Input<IN>
processWatermark
in class AbstractPythonFunctionOperator<OUT>
Exception
Watermark
public <T> AbstractExternalDataStreamPythonFunctionOperator<T> copy(DataStreamPythonFunctionInfo pythonFunctionInfo, TypeInformation<T> outputTypeInfo)
DataStreamPythonFunctionOperator
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.