@Internal public abstract class OneInputPythonFunctionOperator<IN,OUT,UDFIN,UDFOUT> extends AbstractOneInputPythonFunctionOperator<IN,OUT>
OneInputPythonFunctionOperator
is responsible for launching beam runner which will start
a python harness to execute user defined python function.
The operator will buffer the timestamp of input elements in a queue, and set into the produced output element.
Modifier and Type | Field and Description |
---|---|
protected ByteArrayInputStreamWithPos |
bais
Reusable InputStream used to holding the execution results to be deserialized.
|
protected DataInputViewStreamWrapper |
baisWrapper
InputStream Wrapper.
|
protected ByteArrayOutputStreamWithPos |
baos
Reusable OutputStream used to holding the serialized input elements.
|
protected DataOutputViewStreamWrapper |
baosWrapper
OutputStream Wrapper.
|
protected TimestampedCollector<OUT> |
collector |
protected static String |
DATA_STREAM_STATELESS_FUNCTION_URN |
protected Map<String,String> |
jobOptions
The options used to configure the Python worker process.
|
elementCount, maxBundleSize, pythonFunctionRunner
chainingStrategy, latencyStats, LOG, metrics, output, processingTimeService
Constructor and Description |
---|
OneInputPythonFunctionOperator(Configuration config,
TypeInformation<UDFIN> runnerInputTypeInfo,
TypeInformation<UDFOUT> runnerOutputTypeInfo,
DataStreamPythonFunctionInfo pythonFunctionInfo) |
Modifier and Type | Method and Description |
---|---|
PythonFunctionRunner |
createPythonFunctionRunner()
Creates the
PythonFunctionRunner which is responsible for Python user-defined
function execution. |
abstract String |
getCoderUrn() |
String |
getFunctionUrn() |
Map<String,String> |
getInternalParameters() |
PythonEnv |
getPythonEnv()
Returns the
PythonEnv used to create PythonEnvironmentManager.. |
void |
open()
This method is called immediately before any elements are processed, it should contain the
operator's initialization logic, e.g.
|
void |
processElement(StreamRecord<IN> element)
Processes one element that arrived on this input of the
MultipleInputStreamOperator . |
endInput
checkInvokeFinishBundleByCount, close, createPythonEnvironmentManager, dispose, emitResult, emitResults, getConfig, getFlinkMetricContainer, getPythonConfig, invokeFinishBundle, isBundleFinished, prepareSnapshotPreBarrier, processWatermark, setPythonConfig
getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, initializeState, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark1, processWatermark2, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState, snapshotState
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
setKeyContextElement
close, dispose, getMetricGroup, getOperatorID, initializeState, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState
notifyCheckpointAborted, notifyCheckpointComplete
getCurrentKey, setCurrentKey
processLatencyMarker, processWatermark
protected static final String DATA_STREAM_STATELESS_FUNCTION_URN
protected final Map<String,String> jobOptions
protected transient ByteArrayInputStreamWithPos bais
protected transient DataInputViewStreamWrapper baisWrapper
protected transient ByteArrayOutputStreamWithPos baos
protected transient DataOutputViewStreamWrapper baosWrapper
protected transient TimestampedCollector<OUT> collector
public OneInputPythonFunctionOperator(Configuration config, TypeInformation<UDFIN> runnerInputTypeInfo, TypeInformation<UDFOUT> runnerOutputTypeInfo, DataStreamPythonFunctionInfo pythonFunctionInfo)
public void open() throws Exception
AbstractStreamOperator
The default implementation does nothing.
open
in interface StreamOperator<OUT>
open
in class AbstractPythonFunctionOperator<OUT>
Exception
- An exception in this method causes the operator to fail.public PythonFunctionRunner createPythonFunctionRunner() throws Exception
AbstractPythonFunctionOperator
PythonFunctionRunner
which is responsible for Python user-defined
function execution.createPythonFunctionRunner
in class AbstractPythonFunctionOperator<OUT>
Exception
public PythonEnv getPythonEnv()
AbstractPythonFunctionOperator
PythonEnv
used to create PythonEnvironmentManager..getPythonEnv
in class AbstractPythonFunctionOperator<OUT>
public void processElement(StreamRecord<IN> element) throws Exception
Input
MultipleInputStreamOperator
.
This method is guaranteed to not be called concurrently with other methods of the operator.Exception
public String getFunctionUrn()
public abstract String getCoderUrn()
Copyright © 2014–2021 The Apache Software Foundation. All rights reserved.