@Internal public abstract class AbstractPythonStreamAggregateOperator extends AbstractOneInputPythonFunctionOperator<RowData,RowData>
AbstractPythonStreamGroupAggregateOperator
and PythonStreamGroupWindowAggregateOperator
.Modifier and Type | Field and Description |
---|---|
protected ByteArrayInputStreamWithPos |
bais
Reusable InputStream used to holding the execution results to be deserialized.
|
protected DataInputViewStreamWrapper |
baisWrapper
InputStream Wrapper.
|
protected ByteArrayOutputStreamWithPos |
baos
Reusable OutputStream used to holding the serialized input elements.
|
protected DataOutputViewStreamWrapper |
baosWrapper
OutputStream Wrapper.
|
protected RowType |
inputType
The input logical type.
|
protected RowType |
outputType
The output logical type.
|
protected StreamRecordRowDataWrappingCollector |
rowDataWrapper
The collector used to collect records.
|
protected RowType |
userDefinedFunctionInputType
The user-defined function input logical type.
|
protected RowType |
userDefinedFunctionOutputType
The user-defined function output logical type.
|
pythonFunctionRunner
bundleFinishedCallback, config, elementCount, lastFinishBundleTime, maxBundleSize, systemEnvEnabled
chainingStrategy, latencyStats, LOG, metrics, output, processingTimeService
Constructor and Description |
---|
AbstractPythonStreamAggregateOperator(Configuration config,
RowType inputType,
RowType outputType,
PythonAggregateFunctionInfo[] aggregateFunctions,
DataViewSpec[][] dataViewSpecs,
int[] grouping,
int indexOfCountStar,
boolean generateUpdateBefore) |
Modifier and Type | Method and Description |
---|---|
FlinkFnApi.CoderInfoDescriptor |
createInputCoderInfoDescriptor(RowType runnerInputType) |
FlinkFnApi.CoderInfoDescriptor |
createOutputCoderInfoDescriptor(RowType runnerOutType) |
PythonFunctionRunner |
createPythonFunctionRunner()
Creates the
PythonFunctionRunner which is responsible for Python user-defined
function execution. |
abstract RowType |
createUserDefinedFunctionInputType() |
abstract RowType |
createUserDefinedFunctionOutputType() |
Object |
getCurrentKey() |
abstract String |
getFunctionUrn() |
protected RowType |
getKeyType() |
PythonEnv |
getPythonEnv()
Returns the
PythonEnv used to create PythonEnvironmentManager.. |
protected FlinkFnApi.UserDefinedAggregateFunctions |
getUserDefinedFunctionsProto()
Gets the proto representation of the Python user-defined aggregate functions to be executed.
|
void |
open()
This method is called immediately before any elements are processed, it should contain the
operator's initialization logic, e.g. state initialization.
|
void |
processElement(StreamRecord<RowData> element)
Processes one element that arrived on this input of the
MultipleInputStreamOperator . |
abstract void |
processElementInternal(RowData value) |
void |
setCurrentKey(Object key)
As the beam state gRPC service will access the KeyedStateBackend in parallel with this
operator, we must override this method to prevent changing the current key of the
KeyedStateBackend while the beam service is handling requests.
|
endInput
close, createPythonEnvironmentManager, emitResult, emitResults, invokeFinishBundle
checkInvokeFinishBundleByCount, finish, getConfiguration, getFlinkMetricContainer, isBundleFinished, prepareSnapshotPreBarrier, processWatermark
getChainingStrategy, getContainingTask, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark1, processWatermark2, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setChainingStrategy, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState, snapshotState
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
setKeyContextElement
close, finish, getMetricGroup, getOperatorID, initializeState, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState
notifyCheckpointAborted, notifyCheckpointComplete
processLatencyMarker, processWatermark, processWatermarkStatus
hasKeyContext
protected final RowType inputType
protected final RowType outputType
protected transient RowType userDefinedFunctionInputType
protected transient RowType userDefinedFunctionOutputType
protected transient ByteArrayInputStreamWithPos bais
protected transient DataInputViewStreamWrapper baisWrapper
protected transient ByteArrayOutputStreamWithPos baos
protected transient DataOutputViewStreamWrapper baosWrapper
protected transient StreamRecordRowDataWrappingCollector rowDataWrapper
public AbstractPythonStreamAggregateOperator(Configuration config, RowType inputType, RowType outputType, PythonAggregateFunctionInfo[] aggregateFunctions, DataViewSpec[][] dataViewSpecs, int[] grouping, int indexOfCountStar, boolean generateUpdateBefore)
public void open() throws Exception
AbstractStreamOperator
The default implementation does nothing.
open
in interface StreamOperator<RowData>
open
in class AbstractExternalPythonFunctionOperator<RowData>
Exception
- An exception in this method causes the operator to fail.public void processElement(StreamRecord<RowData> element) throws Exception
Input
MultipleInputStreamOperator
.
This method is guaranteed to not be called concurrently with other methods of the operator.Exception
public PythonFunctionRunner createPythonFunctionRunner() throws Exception
AbstractExternalPythonFunctionOperator
PythonFunctionRunner
which is responsible for Python user-defined
function execution.createPythonFunctionRunner
in class AbstractExternalPythonFunctionOperator<RowData>
Exception
public void setCurrentKey(Object key)
setCurrentKey
in interface KeyContext
setCurrentKey
in class AbstractPythonFunctionOperator<RowData>
public Object getCurrentKey()
getCurrentKey
in interface KeyContext
getCurrentKey
in class AbstractStreamOperator<RowData>
public PythonEnv getPythonEnv()
AbstractExternalPythonFunctionOperator
PythonEnv
used to create PythonEnvironmentManager..getPythonEnv
in class AbstractExternalPythonFunctionOperator<RowData>
protected RowType getKeyType()
protected FlinkFnApi.UserDefinedAggregateFunctions getUserDefinedFunctionsProto()
public abstract String getFunctionUrn()
public abstract void processElementInternal(RowData value) throws Exception
Exception
public abstract RowType createUserDefinedFunctionInputType()
public abstract RowType createUserDefinedFunctionOutputType()
public FlinkFnApi.CoderInfoDescriptor createInputCoderInfoDescriptor(RowType runnerInputType)
public FlinkFnApi.CoderInfoDescriptor createOutputCoderInfoDescriptor(RowType runnerOutType)
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.