org.apache.flink.streaming.api.operators.python.process

Class AbstractExternalOneInputPythonFunctionOperator<IN,OUT>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator<OUT>

org.apache.flink.streaming.api.operators.python.process.AbstractExternalPythonFunctionOperator<OUT>

org.apache.flink.streaming.api.operators.python.process.AbstractExternalDataStreamPythonFunctionOperator<OUT>

org.apache.flink.streaming.api.operators.python.process.AbstractExternalOneInputPythonFunctionOperator<IN,OUT>

All Implemented Interfaces:

Serializable, CheckpointListener, ResultTypeQueryable<OUT>, BoundedOneInput, Input<IN>, KeyContext, OneInputStreamOperator<IN,OUT>, DataStreamPythonFunctionOperator<OUT>, SetupableStreamOperator<OUT>, StreamOperator<OUT>, StreamOperatorStateHandler.CheckpointedStreamOperator

Direct Known Subclasses:

ExternalPythonKeyedProcessOperator, ExternalPythonProcessOperator

@Internal
public abstract class AbstractExternalOneInputPythonFunctionOperator<IN,OUT>
extends AbstractExternalDataStreamPythonFunctionOperator<OUT>
implements OneInputStreamOperator<IN,OUT>, BoundedOneInput

AbstractExternalOneInputPythonFunctionOperator is responsible for launching beam runner which will start a python harness to execute user defined python function.

The operator will buffer the timestamp of input elements in a queue, and set into the produced output element.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected ByteArrayOutputStreamWithPos	baos Reusable OutputStream used to holding the serialized input elements.
protected DataOutputViewStreamWrapper	baosWrapper OutputStream Wrapper.

Fields inherited from class org.apache.flink.streaming.api.operators.python.process.AbstractExternalPythonFunctionOperator

pythonFunctionRunner

Fields inherited from class org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator

bundleFinishedCallback, config, elementCount, lastFinishBundleTime, maxBundleSize, systemEnvEnabled

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, latencyStats, LOG, metrics, output, processingTimeService

Constructor Summary

Constructors

Constructor and Description
AbstractExternalOneInputPythonFunctionOperator(Configuration config, DataStreamPythonFunctionInfo pythonFunctionInfo, TypeInformation<IN> inputTypeInfo, TypeInformation<OUT> outputTypeInfo)

Method Summary

Modifier and Type	Method and Description
FlinkFnApi.CoderInfoDescriptor	createInputCoderInfoDescriptor()
FlinkFnApi.CoderInfoDescriptor	createOutputCoderInfoDescriptor()
void	emitResult(Tuple3<String,byte[],Integer> resultTuple) Sends the execution result to the downstream operator.
void	endInput() It is notified that no more data will arrive from the input.
TypeInformation<IN>	getInputTypeInfo()
void	open() This method is called immediately before any elements are processed, it should contain the operator's initialization logic, e.g.
void	processElement(long timestamp, long watermark, Object element)

Methods inherited from class org.apache.flink.streaming.api.operators.python.process.AbstractExternalDataStreamPythonFunctionOperator

addSideOutputTags, createSideOutputCoderDescriptors, getInternalParameters, getOutputTagById, getProducedType, getPythonEnv, getPythonFunctionInfo, getSideOutputTags, getSideOutputTypeSerializerById, setNumPartitions

Methods inherited from class org.apache.flink.streaming.api.operators.python.process.AbstractExternalPythonFunctionOperator

close, createPythonEnvironmentManager, createPythonFunctionRunner, emitResults, invokeFinishBundle

Methods inherited from class org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator

checkInvokeFinishBundleByCount, finish, getConfiguration, getFlinkMetricContainer, isBundleFinished, prepareSnapshotPreBarrier, processWatermark, setCurrentKey

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, initializeState, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, registerCounterOnOutput, reportOrForwardLatencyMarker, setChainingStrategy, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState, snapshotState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

setKeyContextElement

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

close, finish, getMetricGroup, getOperatorID, initializeState, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted, notifyCheckpointComplete

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

getCurrentKey, setCurrentKey

Methods inherited from interface org.apache.flink.streaming.api.operators.Input

processElement, processLatencyMarker, processWatermark, processWatermarkStatus

Methods inherited from interface org.apache.flink.streaming.api.operators.python.DataStreamPythonFunctionOperator

copy

Field Detail

baos

protected transient ByteArrayOutputStreamWithPos baos

Reusable OutputStream used to holding the serialized input elements.

baosWrapper

protected transient DataOutputViewStreamWrapper baosWrapper

OutputStream Wrapper.

Constructor Detail

AbstractExternalOneInputPythonFunctionOperator

public AbstractExternalOneInputPythonFunctionOperator(Configuration config,
                                                      DataStreamPythonFunctionInfo pythonFunctionInfo,
                                                      TypeInformation<IN> inputTypeInfo,
                                                      TypeInformation<OUT> outputTypeInfo)

Method Detail

open

public void open()
          throws Exception

Description copied from class: AbstractStreamOperator

This method is called immediately before any elements are processed, it should contain the operator's initialization logic, e.g. state initialization.

The default implementation does nothing.

Specified by:

open in interface StreamOperator<OUT>

Overrides:

open in class AbstractExternalDataStreamPythonFunctionOperator<OUT>

Throws:

Exception - An exception in this method causes the operator to fail.

endInput

public void endInput()
              throws Exception

Description copied from interface: BoundedOneInput

It is notified that no more data will arrive from the input.

WARNING: It is not safe to use this method to commit any transactions or other side effects! You can use this method to flush any buffered data that can later on be committed e.g. in a CheckpointListener.notifyCheckpointComplete(long).

NOTE: Given it is semantically very similar to the StreamOperator.finish() method. It might be dropped in favour of the other method at some point in time.

Specified by:

endInput in interface BoundedOneInput

Throws:

Exception

emitResult

public void emitResult(Tuple3<String,byte[],Integer> resultTuple)
                throws Exception

Description copied from class: AbstractExternalPythonFunctionOperator

Sends the execution result to the downstream operator.

Specified by:

emitResult in class AbstractExternalPythonFunctionOperator<OUT>

Throws:

Exception

processElement

public void processElement(long timestamp,
                           long watermark,
                           Object element)
                    throws Exception

Throws:

Exception

createInputCoderInfoDescriptor

public FlinkFnApi.CoderInfoDescriptor createInputCoderInfoDescriptor()

createOutputCoderInfoDescriptor

public FlinkFnApi.CoderInfoDescriptor createOutputCoderInfoDescriptor()

getInputTypeInfo

public TypeInformation<IN> getInputTypeInfo()