org.apache.flink.table.runtime.functions.python

Class AbstractPythonStatelessFunctionFlatMap

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.api.common.functions.RichFlatMapFunction<Row,Row>

org.apache.flink.table.runtime.functions.python.AbstractPythonStatelessFunctionFlatMap

All Implemented Interfaces:

Serializable, FlatMapFunction<Row,Row>, Function, RichFunction, ResultTypeQueryable<Row>

Direct Known Subclasses:

AbstractPythonScalarFunctionFlatMap, PythonTableFunctionFlatMap

@Internal
public abstract class AbstractPythonStatelessFunctionFlatMap
extends RichFlatMapFunction<Row,Row>
implements ResultTypeQueryable<Row>

Base Python stateless RichFlatMapFunction used to invoke Python stateless functions for the old planner.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected ByteArrayInputStreamWithPos	bais Reusable InputStream used to holding the execution results to be deserialized.
protected DataInputViewStreamWrapper	baisWrapper InputStream Wrapper.
protected ByteArrayOutputStreamWithPos	baos Reusable OutputStream used to holding the serialized input elements.
protected LinkedBlockingQueue<Row>	forwardedInputQueue The queue holding the input elements for which the execution results have not been received.
protected TypeSerializer<Row>	forwardedInputSerializer The type serializer for the forwarded fields.
protected RowType	inputType The input logical type.
protected Map<String,String>	jobOptions The options used to configure the Python worker process.
protected RowType	outputType The output logical type.
protected PythonFunctionRunner	pythonFunctionRunner The PythonFunctionRunner which is responsible for Python user-defined function execution.
protected Collector<Row>	resultCollector The collector used to collect records.
protected RowType	userDefinedFunctionInputType The user-defined function input logical type.
protected RowType	userDefinedFunctionOutputType The user-defined function output logical type.

Constructor Summary

Constructors

Constructor and Description
AbstractPythonStatelessFunctionFlatMap(Configuration config, RowType inputType, RowType outputType, int[] userDefinedFunctionInputOffsets)

Method Summary

Modifier and Type	Method and Description
abstract void	bufferInput(Row input)
protected void	checkInvokeFinishBundleByCount() Checks whether to invoke finishBundle by elements count.
void	close() Tear-down method for the user code.
protected PythonEnvironmentManager	createPythonEnvironmentManager()
abstract void	emitResult(Tuple2<byte[],Integer> resultTuple)
void	flatMap(Row value, Collector<Row> out) The core method of the FlatMapFunction.
protected FlinkMetricContainer	getFlinkMetricContainer()
abstract int	getForwardedFieldsCount()
protected Row	getFunctionInput(Row element)
abstract String	getFunctionUrn()
abstract String	getInputOutputCoderUrn()
TypeInformation<Row>	getProducedType() Gets the data type (as a TypeInformation) produced by this function or input format.
protected PythonConfig	getPythonConfig()
abstract PythonEnv	getPythonEnv() Returns the PythonEnv used to create PythonEnvironmentManager..
abstract FlinkFnApi.UserDefinedFunctions	getUserDefinedFunctionsProto() Gets the proto representation of the Python user-defined functions to be executed.
protected void	invokeFinishBundle()
void	open(Configuration parameters) Initialization method for the function.
abstract void	processElementInternal(Row value)

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

inputType

protected final RowType inputType

The input logical type.

outputType

protected final RowType outputType

The output logical type.

jobOptions

protected final Map<String,String> jobOptions

The options used to configure the Python worker process.

userDefinedFunctionInputType

protected transient RowType userDefinedFunctionInputType

The user-defined function input logical type.

userDefinedFunctionOutputType

protected transient RowType userDefinedFunctionOutputType

The user-defined function output logical type.

forwardedInputQueue

protected transient LinkedBlockingQueue<Row> forwardedInputQueue

The queue holding the input elements for which the execution results have not been received.

resultCollector

protected transient Collector<Row> resultCollector

The collector used to collect records.

pythonFunctionRunner

protected transient PythonFunctionRunner pythonFunctionRunner

The PythonFunctionRunner which is responsible for Python user-defined function execution.

bais

protected transient ByteArrayInputStreamWithPos bais

Reusable InputStream used to holding the execution results to be deserialized.

baisWrapper

protected transient DataInputViewStreamWrapper baisWrapper

InputStream Wrapper.

forwardedInputSerializer

protected transient TypeSerializer<Row> forwardedInputSerializer

The type serializer for the forwarded fields.

baos

protected transient ByteArrayOutputStreamWithPos baos

Reusable OutputStream used to holding the serialized input elements.

Constructor Detail

AbstractPythonStatelessFunctionFlatMap

public AbstractPythonStatelessFunctionFlatMap(Configuration config,
                                              RowType inputType,
                                              RowType outputType,
                                              int[] userDefinedFunctionInputOffsets)

Method Detail

getPythonConfig

protected PythonConfig getPythonConfig()

open

public void open(Configuration parameters)
          throws Exception

Description copied from interface: RichFunction

Initialization method for the function. It is called before the actual working methods (like map or join) and thus suitable for one time setup work. For functions that are part of an iteration, this method will be invoked at the beginning of each iteration superstep.

The configuration object passed to the function can be used for configuration and initialization. The configuration contains all parameters that were configured on the function in the program composition.

 public class MyFilter extends RichFilterFunction<String> {

     private String searchString;

     public void open(Configuration parameters) {
         this.searchString = parameters.getString("foo");
     }

     public boolean filter(String value) {
         return value.equals(searchString);
     }
 }
 

By default, this method does nothing.

Specified by:

open in interface RichFunction

Overrides:

open in class AbstractRichFunction

Parameters:

parameters - The configuration containing the parameters attached to the contract.

Throws:

Exception - Implementations may forward exceptions, which are caught by the runtime. When the runtime catches an exception, it aborts the task and lets the fail-over logic decide whether to retry the task execution.

See Also:

Configuration

flatMap

public void flatMap(Row value,
                    Collector<Row> out)
             throws Exception

Description copied from interface: FlatMapFunction

The core method of the FlatMapFunction. Takes an element from the input data set and transforms it into zero, one, or more elements.

Specified by:

flatMap in interface FlatMapFunction<Row,Row>

Specified by:

flatMap in class RichFlatMapFunction<Row,Row>

Parameters:

value - The input value.

out - The collector for returning result values.

Throws:

Exception - This method may throw exceptions. Throwing an exception will cause the operation to fail and may trigger recovery.

getProducedType

public TypeInformation<Row> getProducedType()

Description copied from interface: ResultTypeQueryable

Gets the data type (as a TypeInformation) produced by this function or input format.

Specified by:

getProducedType in interface ResultTypeQueryable<Row>

Returns:

The data type produced by this function or input format.

close

public void close()
           throws Exception

Description copied from interface: RichFunction

Tear-down method for the user code. It is called after the last call to the main working methods (e.g. map or join). For functions that are part of an iteration, this method will be invoked after each iteration superstep.

This method can be used for clean up work.

Specified by:

close in interface RichFunction

Overrides:

close in class AbstractRichFunction

Throws:

Exception - Implementations may forward exceptions, which are caught by the runtime. When the runtime catches an exception, it aborts the task and lets the fail-over logic decide whether to retry the task execution.

getPythonEnv

public abstract PythonEnv getPythonEnv()

Returns the PythonEnv used to create PythonEnvironmentManager..

bufferInput

public abstract void bufferInput(Row input)

emitResult

public abstract void emitResult(Tuple2<byte[],Integer> resultTuple)
                         throws Exception

Throws:

Exception

getForwardedFieldsCount

public abstract int getForwardedFieldsCount()

getUserDefinedFunctionsProto

public abstract FlinkFnApi.UserDefinedFunctions getUserDefinedFunctionsProto()

Gets the proto representation of the Python user-defined functions to be executed.

getInputOutputCoderUrn

public abstract String getInputOutputCoderUrn()

getFunctionUrn

public abstract String getFunctionUrn()

processElementInternal

public abstract void processElementInternal(Row value)
                                     throws Exception

Throws:

Exception

checkInvokeFinishBundleByCount

protected void checkInvokeFinishBundleByCount()
                                       throws Exception

Checks whether to invoke finishBundle by elements count. Called in flatMap.

Throws:

Exception

createPythonEnvironmentManager

protected PythonEnvironmentManager createPythonEnvironmentManager()
                                                           throws IOException

Throws:

IOException

getFlinkMetricContainer

protected FlinkMetricContainer getFlinkMetricContainer()

getFunctionInput

protected Row getFunctionInput(Row element)

invokeFinishBundle

protected void invokeFinishBundle()
                           throws Exception

Throws:

Exception