@Internal public abstract class AbstractExternalOneInputPythonFunctionOperator<IN,OUT> extends AbstractExternalDataStreamPythonFunctionOperator<OUT> implements OneInputStreamOperator<IN,OUT>, BoundedOneInput
AbstractExternalOneInputPythonFunctionOperator
is responsible for launching beam runner
which will start a python harness to execute user defined python function.
The operator will buffer the timestamp of input elements in a queue, and set into the produced output element.
Modifier and Type | Field and Description |
---|---|
protected ByteArrayOutputStreamWithPos |
baos
Reusable OutputStream used to holding the serialized input elements.
|
protected DataOutputViewStreamWrapper |
baosWrapper
OutputStream Wrapper.
|
pythonFunctionRunner
bundleFinishedCallback, config, elementCount, lastFinishBundleTime, maxBundleSize, systemEnvEnabled
chainingStrategy, latencyStats, LOG, metrics, output, processingTimeService
Constructor and Description |
---|
AbstractExternalOneInputPythonFunctionOperator(Configuration config,
DataStreamPythonFunctionInfo pythonFunctionInfo,
TypeInformation<IN> inputTypeInfo,
TypeInformation<OUT> outputTypeInfo) |
Modifier and Type | Method and Description |
---|---|
FlinkFnApi.CoderInfoDescriptor |
createInputCoderInfoDescriptor() |
FlinkFnApi.CoderInfoDescriptor |
createOutputCoderInfoDescriptor() |
void |
emitResult(Tuple3<String,byte[],Integer> resultTuple)
Sends the execution result to the downstream operator.
|
void |
endInput()
It is notified that no more data will arrive from the input.
|
TypeInformation<IN> |
getInputTypeInfo() |
void |
open()
This method is called immediately before any elements are processed, it should contain the
operator's initialization logic, e.g. state initialization.
|
void |
processElement(long timestamp,
long watermark,
Object element) |
addSideOutputTags, createSideOutputCoderDescriptors, getInternalParameters, getOutputTagById, getProducedType, getPythonEnv, getPythonFunctionInfo, getSideOutputTags, getSideOutputTypeSerializerById, setNumPartitions
close, createPythonEnvironmentManager, createPythonFunctionRunner, emitResults, invokeFinishBundle
checkInvokeFinishBundleByCount, finish, getConfiguration, getFlinkMetricContainer, isBundleFinished, prepareSnapshotPreBarrier, processWatermark, setCurrentKey
getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setChainingStrategy, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState, snapshotState
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
setKeyContextElement
close, finish, getMetricGroup, getOperatorID, initializeState, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState
notifyCheckpointAborted, notifyCheckpointComplete
getCurrentKey, setCurrentKey
processElement, processLatencyMarker, processWatermark, processWatermarkStatus
copy
hasKeyContext
protected transient ByteArrayOutputStreamWithPos baos
protected transient DataOutputViewStreamWrapper baosWrapper
public AbstractExternalOneInputPythonFunctionOperator(Configuration config, DataStreamPythonFunctionInfo pythonFunctionInfo, TypeInformation<IN> inputTypeInfo, TypeInformation<OUT> outputTypeInfo)
public void open() throws Exception
AbstractStreamOperator
The default implementation does nothing.
open
in interface StreamOperator<OUT>
open
in class AbstractExternalDataStreamPythonFunctionOperator<OUT>
Exception
- An exception in this method causes the operator to fail.public void endInput() throws Exception
BoundedOneInput
WARNING: It is not safe to use this method to commit any transactions or other side
effects! You can use this method to flush any buffered data that can later on be committed
e.g. in a CheckpointListener.notifyCheckpointComplete(long)
.
NOTE: Given it is semantically very similar to the StreamOperator.finish()
method. It might be dropped in favour of the other method at some point in time.
endInput
in interface BoundedOneInput
Exception
public void emitResult(Tuple3<String,byte[],Integer> resultTuple) throws Exception
AbstractExternalPythonFunctionOperator
emitResult
in class AbstractExternalPythonFunctionOperator<OUT>
Exception
public void processElement(long timestamp, long watermark, Object element) throws Exception
Exception
public FlinkFnApi.CoderInfoDescriptor createInputCoderInfoDescriptor()
public FlinkFnApi.CoderInfoDescriptor createOutputCoderInfoDescriptor()
public TypeInformation<IN> getInputTypeInfo()
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.