org.apache.flink.streaming.api.operators.python

Class AbstractPythonFunctionOperator<IN,OUT>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator<IN,OUT>

All Implemented Interfaces:

Serializable, CheckpointListener, BoundedOneInput, KeyContext, OneInputStreamOperator<IN,OUT>, SetupableStreamOperator<OUT>, StreamOperator<OUT>, StreamOperatorStateHandler.CheckpointedStreamOperator, Disposable

Direct Known Subclasses:

AbstractStatelessFunctionOperator

@Internal
public abstract class AbstractPythonFunctionOperator<IN,OUT>
extends AbstractStreamOperator<OUT>
implements OneInputStreamOperator<IN,OUT>, BoundedOneInput

Base class for all stream operators to execute Python functions.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, latencyStats, LOG, metrics, output, processingTimeService

Constructor Summary

Constructors

Constructor and Description
AbstractPythonFunctionOperator(Configuration config)

Method Summary

Modifier and Type	Method and Description
void	close() This method is called after all records have been added to the operators via the methods OneInputStreamOperator.processElement(StreamRecord), or TwoInputStreamOperator.processElement1(StreamRecord) and TwoInputStreamOperator.processElement2(StreamRecord).
protected PythonEnvironmentManager	createPythonEnvironmentManager()
abstract PythonFunctionRunner<IN>	createPythonFunctionRunner() Creates the PythonFunctionRunner which is responsible for Python user-defined function execution.
void	dispose() This method is called at the very end of the operator's life, both in the case of a successful completion of the operation, and in the case of a failure and canceling.
abstract void	emitResults() Sends the execution results to the downstream operator.
void	endInput() It is notified that no more data will arrive on the input.
protected FlinkMetricContainer	getFlinkMetricContainer()
PythonConfig	getPythonConfig()
abstract PythonEnv	getPythonEnv() Returns the PythonEnv used to create PythonEnvironmentManager..
void	open() This method is called immediately before any elements are processed, it should contain the operator's initialization logic, e.g.
void	prepareSnapshotPreBarrier(long checkpointId) This method is called when the operator should do a snapshot, before it emits its own checkpoint barrier.
void	processElement(StreamRecord<IN> element) Processes one element that arrived at this operator.
void	processWatermark(Watermark mark) Processes a Watermark.

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, initializeState, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, numEventTimeTimers, numProcessingTimeTimers, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark1, processWatermark2, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState, snapshotState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

processLatencyMarker

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

getMetricGroup, getOperatorID, initializeState, setKeyContextElement1, setKeyContextElement2, snapshotState

Methods inherited from interface org.apache.flink.runtime.state.CheckpointListener

notifyCheckpointAborted, notifyCheckpointComplete

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

getCurrentKey, setCurrentKey

Constructor Detail

AbstractPythonFunctionOperator

public AbstractPythonFunctionOperator(Configuration config)

Method Detail

getPythonConfig

public PythonConfig getPythonConfig()

open

public void open()
          throws Exception

Description copied from class: AbstractStreamOperator

This method is called immediately before any elements are processed, it should contain the operator's initialization logic, e.g. state initialization.

The default implementation does nothing.

Specified by:

open in interface StreamOperator<OUT>

Overrides:

open in class AbstractStreamOperator<OUT>

Throws:

Exception - An exception in this method causes the operator to fail.

close

public void close()
           throws Exception

Description copied from class: AbstractStreamOperator

This method is called after all records have been added to the operators via the methods OneInputStreamOperator.processElement(StreamRecord), or TwoInputStreamOperator.processElement1(StreamRecord) and TwoInputStreamOperator.processElement2(StreamRecord).

The method is expected to flush all remaining buffered data. Exceptions during this flushing of buffered should be propagated, in order to cause the operation to be recognized asa failed, because the last data items are not processed properly.

Specified by:

close in interface StreamOperator<OUT>

Overrides:

close in class AbstractStreamOperator<OUT>

Throws:

Exception - An exception in this method causes the operator to fail.

dispose

public void dispose()
             throws Exception

Description copied from class: AbstractStreamOperator

This method is called at the very end of the operator's life, both in the case of a successful completion of the operation, and in the case of a failure and canceling.

This method is expected to make a thorough effort to release all resources that the operator has acquired.

Specified by:

dispose in interface StreamOperator<OUT>

Specified by:

dispose in interface Disposable

Overrides:

dispose in class AbstractStreamOperator<OUT>

Throws:

Exception - if something goes wrong during disposal.

endInput

public void endInput()
              throws Exception

Description copied from interface: BoundedOneInput

It is notified that no more data will arrive on the input.

Specified by:

endInput in interface BoundedOneInput

Throws:

Exception

processElement

public void processElement(StreamRecord<IN> element)
                    throws Exception

Description copied from interface: OneInputStreamOperator

Processes one element that arrived at this operator. This method is guaranteed to not be called concurrently with other methods of the operator.

Specified by:

processElement in interface OneInputStreamOperator<IN,OUT>

Throws:

Exception

prepareSnapshotPreBarrier

public void prepareSnapshotPreBarrier(long checkpointId)
                               throws Exception

Description copied from interface: StreamOperator

This method is called when the operator should do a snapshot, before it emits its own checkpoint barrier.

This method is intended not for any actual state persistence, but only for emitting some data before emitting the checkpoint barrier. Operators that maintain some small transient state that is inefficient to checkpoint (especially when it would need to be checkpointed in a re-scalable way) but can simply be sent downstream before the checkpoint. An example are opportunistic pre-aggregation operators, which have small the pre-aggregation state that is frequently flushed downstream.

Important: This method should not be used for any actual state snapshot logic, because it will inherently be within the synchronous part of the operator's checkpoint. If heavy work is done within this method, it will affect latency and downstream checkpoint alignments.

Specified by:

prepareSnapshotPreBarrier in interface StreamOperator<OUT>

Overrides:

prepareSnapshotPreBarrier in class AbstractStreamOperator<OUT>

Parameters:

checkpointId - The ID of the checkpoint.

Throws:

Exception - Throwing an exception here causes the operator to fail and go into recovery.

processWatermark

public void processWatermark(Watermark mark)
                      throws Exception

Description copied from interface: OneInputStreamOperator

Processes a Watermark. This method is guaranteed to not be called concurrently with other methods of the operator.

Specified by:

processWatermark in interface OneInputStreamOperator<IN,OUT>

Overrides:

processWatermark in class AbstractStreamOperator<OUT>

Throws:

Exception

See Also:

Watermark

createPythonFunctionRunner

public abstract PythonFunctionRunner<IN> createPythonFunctionRunner()
                                                             throws Exception

Creates the PythonFunctionRunner which is responsible for Python user-defined function execution.

Throws:

Exception

getPythonEnv

public abstract PythonEnv getPythonEnv()

Returns the PythonEnv used to create PythonEnvironmentManager..

emitResults

public abstract void emitResults()
                          throws IOException

Sends the execution results to the downstream operator.

Throws:

IOException

createPythonEnvironmentManager

protected PythonEnvironmentManager createPythonEnvironmentManager()
                                                           throws IOException

Throws:

IOException

getFlinkMetricContainer

protected FlinkMetricContainer getFlinkMetricContainer()