org.apache.flink.streaming.api.operators.python

Class AbstractPythonFunctionOperator<OUT>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator<OUT>

All Implemented Interfaces:

Serializable, CheckpointListener, KeyContext, KeyContextHandler, SetupableStreamOperator<OUT>, StreamOperator<OUT>, StreamOperatorStateHandler.CheckpointedStreamOperator, YieldingOperator<OUT>

Direct Known Subclasses:

AbstractEmbeddedPythonFunctionOperator, AbstractExternalPythonFunctionOperator, DelegateOperatorTransformation.DelegateOperator

@Internal
public abstract class AbstractPythonFunctionOperator<OUT>
extends AbstractStreamOperator<OUT>

Base class for all stream operators to execute Python functions.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected Runnable	bundleFinishedCallback Callback to be executed after the current bundle was finished.
protected Configuration	config
protected int	elementCount Number of processed elements in the current bundle.
protected long	lastFinishBundleTime Time that the last bundle was finished.
protected int	maxBundleSize Max number of elements to include in a bundle.
protected boolean	systemEnvEnabled

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, lastRecordAttributes1, lastRecordAttributes2, latencyStats, LOG, metrics, output, processingTimeService, stateHandler, stateKeySelector1, stateKeySelector2, timeServiceManager

Constructor Summary

Constructors

Constructor and Description
AbstractPythonFunctionOperator(Configuration config)

Method Summary

Modifier and Type	Method and Description
protected void	checkInvokeFinishBundleByCount() Checks whether to invoke finishBundle by elements count.
void	close() This method is called at the very end of the operator's life, both in the case of a successful completion of the operation, and in the case of a failure and canceling.
protected abstract PythonEnvironmentManager	createPythonEnvironmentManager()
void	finish() This method is called at the end of data processing.
Configuration	getConfiguration() Returns the Configuration.
protected FlinkMetricContainer	getFlinkMetricContainer()
protected abstract void	invokeFinishBundle()
boolean	isBundleFinished() Returns whether the bundle is finished.
void	open() This method is called immediately before any elements are processed, it should contain the operator's initialization logic, e.g. state initialization.
void	prepareSnapshotPreBarrier(long checkpointId) This method is called when the operator should do a snapshot, before it emits its own checkpoint barrier.
void	processWatermark(Watermark mark)
void	setCurrentKey(Object key)

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setChainingStrategy, setKeyContextElement1, setKeyContextElement2, setMailboxExecutor, setProcessingTimeService, setup, snapshotState, snapshotState, useSplittableTimers

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

getOperatorAttributes

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

hasKeyContext

Field Detail

config

protected final Configuration config

systemEnvEnabled

protected transient boolean systemEnvEnabled

maxBundleSize

protected transient int maxBundleSize

Max number of elements to include in a bundle.

elementCount

protected transient int elementCount

Number of processed elements in the current bundle.

lastFinishBundleTime

protected transient long lastFinishBundleTime

Time that the last bundle was finished.

bundleFinishedCallback

protected transient Runnable bundleFinishedCallback

Callback to be executed after the current bundle was finished.

Constructor Detail

AbstractPythonFunctionOperator

public AbstractPythonFunctionOperator(Configuration config)

Method Detail

open

public void open()
          throws Exception

Description copied from class: AbstractStreamOperator

This method is called immediately before any elements are processed, it should contain the operator's initialization logic, e.g. state initialization.

The default implementation does nothing.

Specified by:

open in interface StreamOperator<OUT>

Overrides:

open in class AbstractStreamOperator<OUT>

Throws:

Exception - An exception in this method causes the operator to fail.

finish

public void finish()
            throws Exception

Description copied from interface: StreamOperator

This method is called at the end of data processing.

The method is expected to flush all remaining buffered data. Exceptions during this flushing of buffered data should be propagated, in order to cause the operation to be recognized as failed, because the last data items are not processed properly.

After this method is called, no more records can be produced for the downstream operators.

WARNING: It is not safe to use this method to commit any transactions or other side effects! You can use this method to flush any buffered data that can later on be committed e.g. in a CheckpointListener.notifyCheckpointComplete(long).

NOTE:This method does not need to close any resources. You should release external resources in the StreamOperator.close() method.

Specified by:

finish in interface StreamOperator<OUT>

Overrides:

finish in class AbstractStreamOperator<OUT>

Throws:

Exception - An exception in this method causes the operator to fail.

close

public void close()
           throws Exception

Description copied from interface: StreamOperator

This method is called at the very end of the operator's life, both in the case of a successful completion of the operation, and in the case of a failure and canceling.

This method is expected to make a thorough effort to release all resources that the operator has acquired.

NOTE:It can not emit any records! If you need to emit records at the end of processing, do so in the StreamOperator.finish() method.

Specified by:

close in interface StreamOperator<OUT>

Overrides:

close in class AbstractStreamOperator<OUT>

Throws:

Exception

prepareSnapshotPreBarrier

public void prepareSnapshotPreBarrier(long checkpointId)
                               throws Exception

Description copied from interface: StreamOperator

This method is called when the operator should do a snapshot, before it emits its own checkpoint barrier.

This method is intended not for any actual state persistence, but only for emitting some data before emitting the checkpoint barrier. Operators that maintain some small transient state that is inefficient to checkpoint (especially when it would need to be checkpointed in a re-scalable way) but can simply be sent downstream before the checkpoint. An example are opportunistic pre-aggregation operators, which have small the pre-aggregation state that is frequently flushed downstream.

Important: This method should not be used for any actual state snapshot logic, because it will inherently be within the synchronous part of the operator's checkpoint. If heavy work is done within this method, it will affect latency and downstream checkpoint alignments.

Specified by:

prepareSnapshotPreBarrier in interface StreamOperator<OUT>

Overrides:

prepareSnapshotPreBarrier in class AbstractStreamOperator<OUT>

Parameters:

checkpointId - The ID of the checkpoint.

Throws:

Exception - Throwing an exception here causes the operator to fail and go into recovery.

processWatermark

public void processWatermark(Watermark mark)
                      throws Exception

Overrides:

processWatermark in class AbstractStreamOperator<OUT>

Throws:

Exception

setCurrentKey

public void setCurrentKey(Object key)

Specified by:

setCurrentKey in interface KeyContext

Overrides:

setCurrentKey in class AbstractStreamOperator<OUT>

isBundleFinished

public boolean isBundleFinished()

Returns whether the bundle is finished.

getConfiguration

public Configuration getConfiguration()

Returns the Configuration.

invokeFinishBundle

protected abstract void invokeFinishBundle()
                                    throws Exception

Throws:

Exception

createPythonEnvironmentManager

protected abstract PythonEnvironmentManager createPythonEnvironmentManager()

checkInvokeFinishBundleByCount

protected void checkInvokeFinishBundleByCount()
                                       throws Exception

Checks whether to invoke finishBundle by elements count. Called in processElement.

Throws:

Exception

getFlinkMetricContainer

protected FlinkMetricContainer getFlinkMetricContainer()