Package org.apache.flink.streaming.api.operators.python.process

Class AbstractExternalDataStreamPythonFunctionOperator<OUT>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator<OUT>

org.apache.flink.streaming.api.operators.python.process.AbstractExternalPythonFunctionOperator<OUT>

org.apache.flink.streaming.api.operators.python.process.AbstractExternalDataStreamPythonFunctionOperator<OUT>

All Implemented Interfaces:

Serializable, CheckpointListener, ResultTypeQueryable<OUT>, KeyContext, KeyContextHandler, DataStreamPythonFunctionOperator<OUT>, StreamOperator<OUT>, StreamOperatorStateHandler.CheckpointedStreamOperator, YieldingOperator<OUT>

Direct Known Subclasses:

AbstractExternalOneInputPythonFunctionOperator, AbstractExternalTwoInputPythonFunctionOperator

@Internal
public abstract class AbstractExternalDataStreamPythonFunctionOperator<OUT>
extends AbstractExternalPythonFunctionOperator<OUT>
implements DataStreamPythonFunctionOperator<OUT>

Base class for all Python DataStream operators executed in Python process.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.flink.streaming.api.operators.python.process.AbstractExternalPythonFunctionOperator

pythonFunctionRunner

Fields inherited from class org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator

bundleFinishedCallback, config, elementCount, lastFinishBundleTime, maxBundleSize, systemEnvEnabled

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

lastRecordAttributes1, lastRecordAttributes2, latencyStats, LOG, metrics, output, processingTimeService, stateHandler, stateKeySelector1, stateKeySelector2, timeServiceManager

Constructor Summary

Constructors

Constructor	Description
AbstractExternalDataStreamPythonFunctionOperator(Configuration config, DataStreamPythonFunctionInfo pythonFunctionInfo, TypeInformation<OUT> outputTypeInfo)

Method Summary

Modifier and Type	Method	Description
void	addSideOutputTags(Collection<OutputTag<?>> outputTags)	Add a collection of OutputTags to the operator.
protected Map<String,FlinkFnApi.CoderInfoDescriptor>	createSideOutputCoderDescriptors()
Map<String,String>	getInternalParameters()
protected OutputTag<?>	getOutputTagById(String id)
TypeInformation<OUT>	getProducedType()	Gets the data type (as a TypeInformation) produced by this function or input format.
PythonEnv	getPythonEnv()	Returns the PythonEnv used to create PythonEnvironmentManager..
DataStreamPythonFunctionInfo	getPythonFunctionInfo()	Returns the underlying DataStreamPythonFunctionInfo.
Collection<OutputTag<?>>	getSideOutputTags()	Gets the OutputTags belongs to the operator.
protected TypeSerializer<Row>	getSideOutputTypeSerializerById(String id)
void	open()	This method is called immediately before any elements are processed, it should contain the operator's initialization logic, e.g. state initialization.
void	setNumPartitions(int numPartitions)	Sets the number of partitions.

Methods inherited from class org.apache.flink.streaming.api.operators.python.process.AbstractExternalPythonFunctionOperator

close, createPythonEnvironmentManager, createPythonFunctionRunner, emitResult, emitResults, invokeFinishBundle

Methods inherited from class org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator

checkInvokeFinishBundleByCount, finish, getConfiguration, getFlinkMetricContainer, isBundleFinished, prepareSnapshotPreBarrier, processWatermark, setCurrentKey

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setKeyContextElement1, setKeyContextElement2, setMailboxExecutor, setProcessingTimeService, setup, snapshotState, snapshotState, useSplittableTimers

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.operators.python.DataStreamPythonFunctionOperator

copy

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

hasKeyContext

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

getOperatorAttributes

Constructor Detail

AbstractExternalDataStreamPythonFunctionOperator

public AbstractExternalDataStreamPythonFunctionOperator(Configuration config,
                                                        DataStreamPythonFunctionInfo pythonFunctionInfo,
                                                        TypeInformation<OUT> outputTypeInfo)

Method Detail

open

public void open()
          throws Exception

Description copied from class: AbstractStreamOperator

This method is called immediately before any elements are processed, it should contain the operator's initialization logic, e.g. state initialization.

The default implementation does nothing.

Specified by:

open in interface StreamOperator<OUT>

Overrides:

open in class AbstractExternalPythonFunctionOperator<OUT>

Throws:

Exception - An exception in this method causes the operator to fail.

getPythonEnv

public PythonEnv getPythonEnv()

Description copied from class: AbstractExternalPythonFunctionOperator

Returns the PythonEnv used to create PythonEnvironmentManager..

Specified by:

getPythonEnv in class AbstractExternalPythonFunctionOperator<OUT>

getProducedType

public TypeInformation<OUT> getProducedType()

Description copied from interface: ResultTypeQueryable

Gets the data type (as a TypeInformation) produced by this function or input format.

Specified by:

getProducedType in interface ResultTypeQueryable<OUT>

Returns:

The data type produced by this function or input format.

getPythonFunctionInfo

public DataStreamPythonFunctionInfo getPythonFunctionInfo()

Description copied from interface: DataStreamPythonFunctionOperator

Returns the underlying DataStreamPythonFunctionInfo.

Specified by:

getPythonFunctionInfo in interface DataStreamPythonFunctionOperator<OUT>

setNumPartitions

public void setNumPartitions(int numPartitions)

Description copied from interface: DataStreamPythonFunctionOperator

Sets the number of partitions. This is used for partitionCustom which takes the number of partitions to partition into as input.

Specified by:

setNumPartitions in interface DataStreamPythonFunctionOperator<OUT>

getInternalParameters

public Map<String,String> getInternalParameters()

getOutputTagById

protected OutputTag<?> getOutputTagById(String id)

addSideOutputTags

public void addSideOutputTags(Collection<OutputTag<?>> outputTags)

Description copied from interface: DataStreamPythonFunctionOperator

Add a collection of OutputTags to the operator.

Specified by:

addSideOutputTags in interface DataStreamPythonFunctionOperator<OUT>

getSideOutputTags

public Collection<OutputTag<?>> getSideOutputTags()

Description copied from interface: DataStreamPythonFunctionOperator

Gets the OutputTags belongs to the operator.

Specified by:

getSideOutputTags in interface DataStreamPythonFunctionOperator<OUT>

createSideOutputCoderDescriptors

protected Map<String,FlinkFnApi.CoderInfoDescriptor> createSideOutputCoderDescriptors()

getSideOutputTypeSerializerById

protected TypeSerializer<Row> getSideOutputTypeSerializerById(String id)