Package org.apache.flink.streaming.api.operators.python.process

Class AbstractExternalPythonFunctionOperator<OUT>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator<OUT>

org.apache.flink.streaming.api.operators.python.process.AbstractExternalPythonFunctionOperator<OUT>

All Implemented Interfaces:

Serializable, CheckpointListener, KeyContext, KeyContextHandler, StreamOperator<OUT>, StreamOperatorStateHandler.CheckpointedStreamOperator, YieldingOperator<OUT>

Direct Known Subclasses:

AbstractExternalDataStreamPythonFunctionOperator, AbstractOneInputPythonFunctionOperator

@Internal
public abstract class AbstractExternalPythonFunctionOperator<OUT>
extends AbstractPythonFunctionOperator<OUT>

Abstract class for all stream operators to execute Python functions in external environment.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field	Description
protected PythonFunctionRunner	pythonFunctionRunner	The PythonFunctionRunner which is responsible for Python user-defined function execution.

Fields inherited from class org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator

bundleFinishedCallback, config, elementCount, lastFinishBundleTime, maxBundleSize, systemEnvEnabled

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

lastRecordAttributes1, lastRecordAttributes2, latencyStats, LOG, metrics, output, processingTimeService, stateHandler, stateKeySelector1, stateKeySelector2, timeServiceManager

Constructor Summary

Constructors

Constructor	Description
AbstractExternalPythonFunctionOperator(Configuration config)

Method Summary

Modifier and Type	Method	Description
void	close()	This method is called at the very end of the operator's life, both in the case of a successful completion of the operation, and in the case of a failure and canceling.
protected ProcessPythonEnvironmentManager	createPythonEnvironmentManager()
abstract PythonFunctionRunner	createPythonFunctionRunner()	Creates the PythonFunctionRunner which is responsible for Python user-defined function execution.
abstract void	emitResult(Tuple3<String,byte[],Integer> resultTuple)	Sends the execution result to the downstream operator.
protected void	emitResults()
abstract PythonEnv	getPythonEnv()	Returns the PythonEnv used to create PythonEnvironmentManager..
protected void	invokeFinishBundle()
void	open()	This method is called immediately before any elements are processed, it should contain the operator's initialization logic, e.g. state initialization.

Methods inherited from class org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator

checkInvokeFinishBundleByCount, finish, getConfiguration, getFlinkMetricContainer, isBundleFinished, prepareSnapshotPreBarrier, processWatermark, setCurrentKey

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setKeyContextElement1, setKeyContextElement2, setMailboxExecutor, setProcessingTimeService, setup, snapshotState, snapshotState, useSplittableTimers

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

hasKeyContext

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

getOperatorAttributes

Field Detail

pythonFunctionRunner

protected transient PythonFunctionRunner pythonFunctionRunner

The PythonFunctionRunner which is responsible for Python user-defined function execution.

Constructor Detail

AbstractExternalPythonFunctionOperator

public AbstractExternalPythonFunctionOperator(Configuration config)

Method Detail

open

public void open()
          throws Exception

Description copied from class: AbstractStreamOperator

This method is called immediately before any elements are processed, it should contain the operator's initialization logic, e.g. state initialization.

The default implementation does nothing.

Specified by:

open in interface StreamOperator<OUT>

Overrides:

open in class AbstractPythonFunctionOperator<OUT>

Throws:

Exception - An exception in this method causes the operator to fail.

close

public void close()
           throws Exception

Description copied from interface: StreamOperator

This method is called at the very end of the operator's life, both in the case of a successful completion of the operation, and in the case of a failure and canceling.

This method is expected to make a thorough effort to release all resources that the operator has acquired.

NOTE:It can not emit any records! If you need to emit records at the end of processing, do so in the StreamOperator.finish() method.

Specified by:

close in interface StreamOperator<OUT>

Overrides:

close in class AbstractPythonFunctionOperator<OUT>

Throws:

Exception

invokeFinishBundle

protected void invokeFinishBundle()
                           throws Exception

Specified by:

invokeFinishBundle in class AbstractPythonFunctionOperator<OUT>

Throws:

Exception

createPythonEnvironmentManager

protected ProcessPythonEnvironmentManager createPythonEnvironmentManager()

Specified by:

createPythonEnvironmentManager in class AbstractPythonFunctionOperator<OUT>

emitResults

protected void emitResults()
                    throws Exception

Throws:

Exception

getPythonEnv

public abstract PythonEnv getPythonEnv()

Returns the PythonEnv used to create PythonEnvironmentManager..

emitResult

public abstract void emitResult(Tuple3<String,byte[],Integer> resultTuple)
                         throws Exception

Sends the execution result to the downstream operator.

Throws:

Exception

createPythonFunctionRunner

public abstract PythonFunctionRunner createPythonFunctionRunner()
                                                         throws Exception

Creates the PythonFunctionRunner which is responsible for Python user-defined function execution.

Throws:

Exception