org.apache.flink.streaming.api.operators.python.process

Class AbstractExternalTwoInputPythonFunctionOperator<IN1,IN2,OUT>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator<OUT>

org.apache.flink.streaming.api.operators.python.process.AbstractExternalPythonFunctionOperator<OUT>

org.apache.flink.streaming.api.operators.python.process.AbstractExternalDataStreamPythonFunctionOperator<OUT>

org.apache.flink.streaming.api.operators.python.process.AbstractExternalTwoInputPythonFunctionOperator<IN1,IN2,OUT>

All Implemented Interfaces:

Serializable, CheckpointListener, ResultTypeQueryable<OUT>, BoundedMultiInput, KeyContext, KeyContextHandler, DataStreamPythonFunctionOperator<OUT>, SetupableStreamOperator<OUT>, StreamOperator<OUT>, StreamOperatorStateHandler.CheckpointedStreamOperator, TwoInputStreamOperator<IN1,IN2,OUT>

Direct Known Subclasses:

ExternalPythonCoProcessOperator, ExternalPythonKeyedCoProcessOperator

@Internal
public abstract class AbstractExternalTwoInputPythonFunctionOperator<IN1,IN2,OUT>
extends AbstractExternalDataStreamPythonFunctionOperator<OUT>
implements TwoInputStreamOperator<IN1,IN2,OUT>, BoundedMultiInput

AbstractExternalTwoInputPythonFunctionOperator is responsible for launching beam runner which will start a python harness to execute two-input user defined python function.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected ByteArrayOutputStreamWithPos	baos
protected DataOutputViewStreamWrapper	baosWrapper

Fields inherited from class org.apache.flink.streaming.api.operators.python.process.AbstractExternalPythonFunctionOperator

pythonFunctionRunner

Fields inherited from class org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator

bundleFinishedCallback, config, elementCount, lastFinishBundleTime, maxBundleSize, systemEnvEnabled

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, latencyStats, LOG, metrics, output, processingTimeService

Constructor Summary

Constructors

Constructor and Description
AbstractExternalTwoInputPythonFunctionOperator(Configuration config, DataStreamPythonFunctionInfo pythonFunctionInfo, TypeInformation<IN1> inputTypeInfo1, TypeInformation<IN2> inputTypeInfo2, TypeInformation<OUT> outputTypeInfo)

Method Summary

Modifier and Type	Method and Description
FlinkFnApi.CoderInfoDescriptor	createInputCoderInfoDescriptor()
FlinkFnApi.CoderInfoDescriptor	createOutputCoderInfoDescriptor()
void	emitResult(Tuple3<String,byte[],Integer> resultTuple) Sends the execution result to the downstream operator.
void	endInput(int inputId) It is notified that no more data will arrive from the input identified by the inputId.
protected TypeInformation<IN1>	getLeftInputType()
protected TypeInformation<IN2>	getRightInputType()
void	open() This method is called immediately before any elements are processed, it should contain the operator's initialization logic, e.g.
void	processElement(boolean isLeft, long timestamp, long watermark, Object element)

Methods inherited from class org.apache.flink.streaming.api.operators.python.process.AbstractExternalDataStreamPythonFunctionOperator

addSideOutputTags, createSideOutputCoderDescriptors, getInternalParameters, getOutputTagById, getProducedType, getPythonEnv, getPythonFunctionInfo, getSideOutputTags, getSideOutputTypeSerializerById, setNumPartitions

Methods inherited from class org.apache.flink.streaming.api.operators.python.process.AbstractExternalPythonFunctionOperator

close, createPythonEnvironmentManager, createPythonFunctionRunner, emitResults, invokeFinishBundle

Methods inherited from class org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator

checkInvokeFinishBundleByCount, finish, getConfiguration, getFlinkMetricContainer, isBundleFinished, prepareSnapshotPreBarrier, processWatermark, setCurrentKey

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setChainingStrategy, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState, snapshotState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.operators.TwoInputStreamOperator

processElement1, processElement2, processLatencyMarker1, processLatencyMarker2, processWatermark1, processWatermark2, processWatermarkStatus1, processWatermarkStatus2

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

close, finish, getMetricGroup, getOperatorID, initializeState, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted, notifyCheckpointComplete

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

getCurrentKey, setCurrentKey

Methods inherited from interface org.apache.flink.streaming.api.operators.python.DataStreamPythonFunctionOperator

copy

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

hasKeyContext

Field Detail

baos

protected transient ByteArrayOutputStreamWithPos baos

baosWrapper

protected transient DataOutputViewStreamWrapper baosWrapper

Constructor Detail

AbstractExternalTwoInputPythonFunctionOperator

public AbstractExternalTwoInputPythonFunctionOperator(Configuration config,
                                                      DataStreamPythonFunctionInfo pythonFunctionInfo,
                                                      TypeInformation<IN1> inputTypeInfo1,
                                                      TypeInformation<IN2> inputTypeInfo2,
                                                      TypeInformation<OUT> outputTypeInfo)

Method Detail

open

public void open()
          throws Exception

Description copied from class: AbstractStreamOperator

This method is called immediately before any elements are processed, it should contain the operator's initialization logic, e.g. state initialization.

The default implementation does nothing.

Specified by:

open in interface StreamOperator<OUT>

Overrides:

open in class AbstractExternalDataStreamPythonFunctionOperator<OUT>

Throws:

Exception - An exception in this method causes the operator to fail.

endInput

public void endInput(int inputId)
              throws Exception

Description copied from interface: BoundedMultiInput

It is notified that no more data will arrive from the input identified by the inputId. The inputId is numbered starting from 1, and `1` indicates the first input.

WARNING: It is not safe to use this method to commit any transactions or other side effects! You can use this method to e.g. flush data buffered for the given input or implement an ordered reading from multiple inputs via InputSelectable.

Specified by:

endInput in interface BoundedMultiInput

Throws:

Exception

emitResult

public void emitResult(Tuple3<String,byte[],Integer> resultTuple)
                throws Exception

Description copied from class: AbstractExternalPythonFunctionOperator

Sends the execution result to the downstream operator.

Specified by:

emitResult in class AbstractExternalPythonFunctionOperator<OUT>

Throws:

Exception

processElement

public void processElement(boolean isLeft,
                           long timestamp,
                           long watermark,
                           Object element)
                    throws Exception

Throws:

Exception

createInputCoderInfoDescriptor

public FlinkFnApi.CoderInfoDescriptor createInputCoderInfoDescriptor()

createOutputCoderInfoDescriptor

public FlinkFnApi.CoderInfoDescriptor createOutputCoderInfoDescriptor()

getLeftInputType

protected TypeInformation<IN1> getLeftInputType()

getRightInputType

protected TypeInformation<IN2> getRightInputType()