IN
- The data type of the input data set.OUT
- The data type of the returned data set.@Deprecated @Public public abstract class SingleInputUdfOperator<IN,OUT,O extends SingleInputUdfOperator<IN,OUT,O>> extends SingleInputOperator<IN,OUT,O> implements UdfOperator<O>
RichMapFunction
or RichReduceFunction
).
This class encapsulates utilities for the UDFs, such as broadcast variables, parameterization through configuration objects, and semantic properties.
minResources, name, parallelism, preferredResources
Modifier | Constructor and Description |
---|---|
protected |
SingleInputUdfOperator(DataSet<IN> input,
TypeInformation<OUT> resultType)
Deprecated.
Creates a new operators with the given data set as input.
|
Modifier and Type | Method and Description |
---|---|
protected SingleInputSemanticProperties |
extractSemanticAnnotations(Class<?> udfClass)
Deprecated.
|
protected boolean |
getAnalyzedUdfSemanticsFlag()
Deprecated.
|
Map<String,DataSet<?>> |
getBroadcastSets()
Deprecated.
Gets the broadcast sets (name and data set) that have been added to context of the UDF.
|
protected abstract Function |
getFunction()
Deprecated.
|
Configuration |
getParameters()
Deprecated.
Gets the configuration parameters that will be passed to the UDF's open method
AbstractRichFunction.open(Configuration) . |
SingleInputSemanticProperties |
getSemanticProperties()
Deprecated.
Gets the semantic properties that have been set for the user-defined functions (UDF).
|
O |
returns(Class<OUT> typeClass)
Deprecated.
Adds a type information hint about the return type of this operator.
|
O |
returns(TypeHint<OUT> typeHint)
Deprecated.
Adds a type information hint about the return type of this operator.
|
O |
returns(TypeInformation<OUT> typeInfo)
Deprecated.
Adds a type information hint about the return type of this operator.
|
protected void |
setAnalyzedUdfSemanticsFlag()
Deprecated.
|
void |
setSemanticProperties(SingleInputSemanticProperties properties)
Deprecated.
Sets the semantic properties for the user-defined function (UDF).
|
protected boolean |
udfWithForwardedFieldsAnnotation(Class<?> udfClass)
Deprecated.
|
O |
withBroadcastSet(DataSet<?> data,
String name)
Deprecated.
Adds a certain data set as a broadcast set to this operator.
|
O |
withForwardedFields(String... forwardedFields)
Deprecated.
Adds semantic information about forwarded fields of the user-defined function.
|
O |
withParameters(Configuration parameters)
Deprecated.
Sets the configuration parameters for the UDF.
|
getInput, getInputType, translateToDataFlow
getMinResources, getName, getParallelism, getPreferredResources, getResultType, name, setParallelism
aggregate, checkSameExecutionContext, clean, coGroup, collect, combineGroup, count, cross, crossWithHuge, crossWithTiny, distinct, distinct, distinct, distinct, fillInType, filter, first, flatMap, fullOuterJoin, fullOuterJoin, getExecutionEnvironment, getType, groupBy, groupBy, groupBy, iterate, iterateDelta, join, join, joinWithHuge, joinWithTiny, leftOuterJoin, leftOuterJoin, map, mapPartition, max, maxBy, min, minBy, output, partitionByHash, partitionByHash, partitionByHash, partitionByRange, partitionByRange, partitionByRange, partitionCustom, partitionCustom, partitionCustom, print, print, printOnTaskManager, printToErr, printToErr, project, rebalance, reduce, reduceGroup, rightOuterJoin, rightOuterJoin, runOperation, sortPartition, sortPartition, sortPartition, sum, union, write, write, writeAsCsv, writeAsCsv, writeAsCsv, writeAsCsv, writeAsFormattedText, writeAsFormattedText, writeAsText, writeAsText
protected SingleInputUdfOperator(DataSet<IN> input, TypeInformation<OUT> resultType)
input
- The data set that is the input to the operator.resultType
- The type of the elements in the resulting data set.protected abstract Function getFunction()
public O withParameters(Configuration parameters)
UdfOperator
AbstractRichFunction.open(Configuration)
method.withParameters
in interface UdfOperator<O extends SingleInputUdfOperator<IN,OUT,O>>
parameters
- The configuration parameters for the UDF.public O withBroadcastSet(DataSet<?> data, String name)
UdfOperator
RuntimeContext.getBroadcastVariable(String)
.
The runtime context itself is available in all UDFs via AbstractRichFunction.getRuntimeContext()
.
withBroadcastSet
in interface UdfOperator<O extends SingleInputUdfOperator<IN,OUT,O>>
data
- The data set to be broadcast.name
- The name under which the broadcast data set retrieved.public O withForwardedFields(String... forwardedFields)
Fields that are forwarded at the same position are specified by their position. The
specified position must be valid for the input and output data type and have the same type.
For example withForwardedFields("f2")
declares that the third field of a Java
input tuple is copied to the third field of an output tuple.
Fields which are unchanged copied to another position in the output are declared by
specifying the source field reference in the input and the target field reference in the
output. withForwardedFields("f0->f2")
denotes that the first field of the Java input
tuple is unchanged copied to the third field of the Java output tuple. When using a wildcard
("*") ensure that the number of declared fields and their types in input and output type
match.
Multiple forwarded fields can be annotated in one (withForwardedFields("f2; f3->f0;
f4")
) or separate Strings (withForwardedFields("f2", "f3->f0", "f4")
). Please refer
to the JavaDoc of Function
or Flink's
documentation for details on field references such as nested fields and wildcard.
It is not possible to override existing semantic information about forwarded fields which
was for example added by a FunctionAnnotation.ForwardedFields
class annotation.
NOTE: Adding semantic information for functions is optional! If used correctly, semantic information can help the Flink optimizer to generate more efficient execution plans. However, incorrect semantic information can cause the optimizer to generate incorrect execution plans which compute wrong results! So be careful when adding semantic information.
forwardedFields
- A list of field forward expressions.FunctionAnnotation
,
FunctionAnnotation.ForwardedFields
public O returns(Class<OUT> typeClass)
Classes can be used as type hints for non-generic types (classes without generic
parameters), but not for generic types like for example Tuples. For those generic types,
please use the returns(TypeHint)
method.
Use this method the following way:
DataSet<String[]> result =
data.flatMap(new FunctionWithNonInferrableReturnType())
.returns(String[].class);
typeClass
- The class of the returned data type.public O returns(TypeHint<OUT> typeHint)
Use this method the following way:
DataSet<Tuple2<String, Double>> result =
data.flatMap(new FunctionWithNonInferrableReturnType())
.returns(new TypeHint<Tuple2<String, Double>>(){});
typeHint
- The type hint for the returned data type.public O returns(TypeInformation<OUT> typeInfo)
In most cases, the methods returns(Class)
and returns(TypeHint)
are
preferable.
typeInfo
- The type information for the returned data type.@Internal public Map<String,DataSet<?>> getBroadcastSets()
UdfOperator
UdfOperator.withBroadcastSet(DataSet, String)
.getBroadcastSets
in interface UdfOperator<O extends SingleInputUdfOperator<IN,OUT,O>>
public Configuration getParameters()
UdfOperator
AbstractRichFunction.open(Configuration)
. The
configuration is set via the UdfOperator.withParameters(Configuration)
method.getParameters
in interface UdfOperator<O extends SingleInputUdfOperator<IN,OUT,O>>
@Internal public SingleInputSemanticProperties getSemanticProperties()
UdfOperator
getSemanticProperties
in interface UdfOperator<O extends SingleInputUdfOperator<IN,OUT,O>>
@Internal public void setSemanticProperties(SingleInputSemanticProperties properties)
UdfOperator.getSemanticProperties()
.properties
- The semantic properties for the UDF.UdfOperator.getSemanticProperties()
protected boolean getAnalyzedUdfSemanticsFlag()
protected void setAnalyzedUdfSemanticsFlag()
protected SingleInputSemanticProperties extractSemanticAnnotations(Class<?> udfClass)
protected boolean udfWithForwardedFieldsAnnotation(Class<?> udfClass)
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.