IN
- Type of the input elements.OUT
- Type of the output elements.UDFIN
- Type of the UDF input type.@Internal public abstract class AbstractStatelessFunctionOperator<IN,OUT,UDFIN> extends AbstractPythonFunctionOperator<IN,OUT>
Modifier and Type | Class and Description |
---|---|
static class |
AbstractStatelessFunctionOperator.StreamRecordCRowWrappingCollector
The collector is used to convert a
Row to a CRow . |
static class |
AbstractStatelessFunctionOperator.StreamRecordRowDataWrappingCollector
The collector is used to convert a
RowData to a StreamRecord . |
Modifier and Type | Field and Description |
---|---|
protected ByteArrayInputStreamWithPos |
bais
Reusable InputStream used to holding the execution results to be deserialized.
|
protected DataInputViewStreamWrapper |
baisWrapper
InputStream Wrapper.
|
protected LinkedBlockingQueue<IN> |
forwardedInputQueue
The queue holding the input elements for which the execution results have not been received.
|
protected RowType |
inputType
The input logical type.
|
protected RowType |
outputType
The output logical type.
|
protected int[] |
userDefinedFunctionInputOffsets
The offsets of user-defined function inputs.
|
protected RowType |
userDefinedFunctionInputType
The user-defined function input logical type.
|
protected RowType |
userDefinedFunctionOutputType
The user-defined function output logical type.
|
protected LinkedBlockingQueue<byte[]> |
userDefinedFunctionResultQueue
The queue holding the user-defined function execution results.
|
chainingStrategy, latencyStats, LOG, metrics, output, processingTimeService
Constructor and Description |
---|
AbstractStatelessFunctionOperator(Configuration config,
RowType inputType,
RowType outputType,
int[] userDefinedFunctionInputOffsets) |
Modifier and Type | Method and Description |
---|---|
abstract void |
bufferInput(IN input)
Buffers the specified input, it will be used to construct the operator result together with
the user-defined function execution result.
|
PythonFunctionRunner<IN> |
createPythonFunctionRunner()
Creates the
PythonFunctionRunner which is responsible for Python user-defined
function execution. |
abstract PythonFunctionRunner<UDFIN> |
createPythonFunctionRunner(org.apache.beam.sdk.fn.data.FnDataReceiver<byte[]> resultReceiver,
PythonEnvironmentManager pythonEnvironmentManager,
Map<String,String> jobOptions) |
abstract UDFIN |
getFunctionInput(IN element) |
void |
open()
This method is called immediately before any elements are processed, it should contain the
operator's initialization logic, e.g.
|
void |
processElement(StreamRecord<IN> element)
Processes one element that arrived at this operator.
|
close, createPythonEnvironmentManager, dispose, emitResults, endInput, getFlinkMetricContainer, getPythonConfig, getPythonEnv, prepareSnapshotPreBarrier, processWatermark
getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, initializeState, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, numEventTimeTimers, numProcessingTimeTimers, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark1, processWatermark2, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState, snapshotState
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
processLatencyMarker
getMetricGroup, getOperatorID, initializeState, setKeyContextElement1, setKeyContextElement2, snapshotState
notifyCheckpointAborted, notifyCheckpointComplete
getCurrentKey, setCurrentKey
protected final RowType inputType
protected final RowType outputType
protected final int[] userDefinedFunctionInputOffsets
protected transient RowType userDefinedFunctionInputType
protected transient RowType userDefinedFunctionOutputType
protected transient LinkedBlockingQueue<IN> forwardedInputQueue
protected transient LinkedBlockingQueue<byte[]> userDefinedFunctionResultQueue
protected transient ByteArrayInputStreamWithPos bais
protected transient DataInputViewStreamWrapper baisWrapper
public AbstractStatelessFunctionOperator(Configuration config, RowType inputType, RowType outputType, int[] userDefinedFunctionInputOffsets)
public void open() throws Exception
AbstractStreamOperator
The default implementation does nothing.
open
in interface StreamOperator<OUT>
open
in class AbstractPythonFunctionOperator<IN,OUT>
Exception
- An exception in this method causes the operator to fail.public void processElement(StreamRecord<IN> element) throws Exception
OneInputStreamOperator
processElement
in interface OneInputStreamOperator<IN,OUT>
processElement
in class AbstractPythonFunctionOperator<IN,OUT>
Exception
public PythonFunctionRunner<IN> createPythonFunctionRunner() throws IOException
AbstractPythonFunctionOperator
PythonFunctionRunner
which is responsible for Python user-defined
function execution.createPythonFunctionRunner
in class AbstractPythonFunctionOperator<IN,OUT>
IOException
public abstract void bufferInput(IN input)
public abstract PythonFunctionRunner<UDFIN> createPythonFunctionRunner(org.apache.beam.sdk.fn.data.FnDataReceiver<byte[]> resultReceiver, PythonEnvironmentManager pythonEnvironmentManager, Map<String,String> jobOptions)
Copyright © 2014–2021 The Apache Software Foundation. All rights reserved.