@PublicEvolving public abstract class AbstractRuntimeUDFContext extends Object implements RuntimeContext
RuntimeContext
, created by runtime UDF operators.Constructor and Description |
---|
AbstractRuntimeUDFContext(TaskInfo taskInfo,
ClassLoader userCodeClassLoader,
ExecutionConfig executionConfig,
Map<String,Accumulator<?,?>> accumulators,
Map<String,Future<Path>> cpTasks,
MetricGroup metrics) |
Modifier and Type | Method and Description |
---|---|
<V,A extends Serializable> |
addAccumulator(String name,
Accumulator<V,A> accumulator)
Add this accumulator.
|
<V,A extends Serializable> |
getAccumulator(String name)
Get an existing accumulator object.
|
<IN,ACC,OUT> |
getAggregatingState(AggregatingStateDescriptor<IN,ACC,OUT> stateProperties)
Gets a handle to the system's key/value aggregating state.
|
Map<String,Accumulator<?,?>> |
getAllAccumulators()
Returns a map of all registered accumulators for this task.
|
int |
getAttemptNumber()
Gets the attempt number of this parallel subtask.
|
DistributedCache |
getDistributedCache()
Returns the
DistributedCache to get the local temporary file copies of files otherwise not
locally accessible. |
DoubleCounter |
getDoubleCounter(String name)
Convenience function to create a counter object for doubles.
|
ExecutionConfig |
getExecutionConfig()
Returns the
ExecutionConfig for the currently executing
job. |
<T,ACC> FoldingState<T,ACC> |
getFoldingState(FoldingStateDescriptor<T,ACC> stateProperties)
Deprecated.
|
Histogram |
getHistogram(String name)
Convenience function to create a counter object for histograms.
|
int |
getIndexOfThisSubtask()
Gets the number of this parallel subtask.
|
IntCounter |
getIntCounter(String name)
Convenience function to create a counter object for integers.
|
<T> ListState<T> |
getListState(ListStateDescriptor<T> stateProperties)
Gets a handle to the system's key/value list state.
|
LongCounter |
getLongCounter(String name)
Convenience function to create a counter object for longs.
|
<UK,UV> MapState<UK,UV> |
getMapState(MapStateDescriptor<UK,UV> stateProperties)
Gets a handle to the system's key/value map state.
|
int |
getMaxNumberOfParallelSubtasks()
Gets the number of max-parallelism with which the parallel task runs.
|
MetricGroup |
getMetricGroup()
Returns the metric group for this parallel subtask.
|
int |
getNumberOfParallelSubtasks()
Gets the parallelism with which the parallel task runs.
|
<T> ReducingState<T> |
getReducingState(ReducingStateDescriptor<T> stateProperties)
Gets a handle to the system's key/value reducing state.
|
<T> ValueState<T> |
getState(ValueStateDescriptor<T> stateProperties)
Gets a handle to the system's key/value state.
|
String |
getTaskName()
Returns the name of the task in which the UDF runs, as assigned during plan construction.
|
String |
getTaskNameWithSubtasks()
Returns the name of the task, appended with the subtask indicator, such as "MyTask (3/6)",
where 3 would be (
RuntimeContext.getIndexOfThisSubtask() + 1), and 6 would be
RuntimeContext.getNumberOfParallelSubtasks() . |
ClassLoader |
getUserCodeClassLoader()
Gets the ClassLoader to load classes that were are not in system's classpath, but are part of the
jar file of a user job.
|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
getBroadcastVariable, getBroadcastVariableWithInitializer, hasBroadcastVariable
public AbstractRuntimeUDFContext(TaskInfo taskInfo, ClassLoader userCodeClassLoader, ExecutionConfig executionConfig, Map<String,Accumulator<?,?>> accumulators, Map<String,Future<Path>> cpTasks, MetricGroup metrics)
public ExecutionConfig getExecutionConfig()
RuntimeContext
ExecutionConfig
for the currently executing
job.getExecutionConfig
in interface RuntimeContext
public String getTaskName()
RuntimeContext
getTaskName
in interface RuntimeContext
public int getNumberOfParallelSubtasks()
RuntimeContext
getNumberOfParallelSubtasks
in interface RuntimeContext
public int getMaxNumberOfParallelSubtasks()
RuntimeContext
getMaxNumberOfParallelSubtasks
in interface RuntimeContext
public int getIndexOfThisSubtask()
RuntimeContext
RuntimeContext.getNumberOfParallelSubtasks()
).getIndexOfThisSubtask
in interface RuntimeContext
public MetricGroup getMetricGroup()
RuntimeContext
getMetricGroup
in interface RuntimeContext
public int getAttemptNumber()
RuntimeContext
getAttemptNumber
in interface RuntimeContext
public String getTaskNameWithSubtasks()
RuntimeContext
RuntimeContext.getIndexOfThisSubtask()
+ 1), and 6 would be
RuntimeContext.getNumberOfParallelSubtasks()
.getTaskNameWithSubtasks
in interface RuntimeContext
public IntCounter getIntCounter(String name)
RuntimeContext
getIntCounter
in interface RuntimeContext
public LongCounter getLongCounter(String name)
RuntimeContext
getLongCounter
in interface RuntimeContext
public Histogram getHistogram(String name)
RuntimeContext
getHistogram
in interface RuntimeContext
public DoubleCounter getDoubleCounter(String name)
RuntimeContext
getDoubleCounter
in interface RuntimeContext
public <V,A extends Serializable> void addAccumulator(String name, Accumulator<V,A> accumulator)
RuntimeContext
addAccumulator
in interface RuntimeContext
public <V,A extends Serializable> Accumulator<V,A> getAccumulator(String name)
RuntimeContext
getAccumulator
in interface RuntimeContext
public Map<String,Accumulator<?,?>> getAllAccumulators()
RuntimeContext
getAllAccumulators
in interface RuntimeContext
public ClassLoader getUserCodeClassLoader()
RuntimeContext
getUserCodeClassLoader
in interface RuntimeContext
public DistributedCache getDistributedCache()
RuntimeContext
DistributedCache
to get the local temporary file copies of files otherwise not
locally accessible.getDistributedCache
in interface RuntimeContext
@PublicEvolving public <T> ValueState<T> getState(ValueStateDescriptor<T> stateProperties)
RuntimeContext
Because the scope of each value is the key of the currently processed element, and the elements are distributed by the Flink runtime, the system can transparently scale out and redistribute the state and KeyedStream.
The following code example shows how to implement a continuous counter that counts how many times elements of a certain key occur, and emits an updated count for that element on each occurrence.
DataStream<MyType> stream = ...;
KeyedStream<MyType> keyedStream = stream.keyBy("id");
keyedStream.map(new RichMapFunction<MyType, Tuple2<MyType, Long>>() {
private ValueState<Long> count;
public void open(Configuration cfg) {
state = getRuntimeContext().getState(
new ValueStateDescriptor<Long>("count", LongSerializer.INSTANCE, 0L));
}
public Tuple2<MyType, Long> map(MyType value) {
long count = state.value() + 1;
state.update(value);
return new Tuple2<>(value, count);
}
});
getState
in interface RuntimeContext
T
- The type of value stored in the state.stateProperties
- The descriptor defining the properties of the stats.@PublicEvolving public <T> ListState<T> getListState(ListStateDescriptor<T> stateProperties)
RuntimeContext
RuntimeContext.getState(ValueStateDescriptor)
, but is optimized for state that
holds lists. One can add elements to the list, or retrieve the list as a whole.
This state is only accessible if the function is executed on a KeyedStream.
DataStream<MyType> stream = ...;
KeyedStream<MyType> keyedStream = stream.keyBy("id");
keyedStream.map(new RichFlatMapFunction<MyType, List<MyType>>() {
private ListState<MyType> state;
public void open(Configuration cfg) {
state = getRuntimeContext().getListState(
new ListStateDescriptor<>("myState", MyType.class));
}
public void flatMap(MyType value, Collector<MyType> out) {
if (value.isDivider()) {
for (MyType t : state.get()) {
out.collect(t);
}
} else {
state.add(value);
}
}
});
getListState
in interface RuntimeContext
T
- The type of value stored in the state.stateProperties
- The descriptor defining the properties of the stats.@PublicEvolving public <T> ReducingState<T> getReducingState(ReducingStateDescriptor<T> stateProperties)
RuntimeContext
RuntimeContext.getState(ValueStateDescriptor)
, but is optimized for state that
aggregates values.
This state is only accessible if the function is executed on a KeyedStream.
DataStream<MyType> stream = ...;
KeyedStream<MyType> keyedStream = stream.keyBy("id");
keyedStream.map(new RichMapFunction<MyType, List<MyType>>() {
private ReducingState<Long> state;
public void open(Configuration cfg) {
state = getRuntimeContext().getReducingState(
new ReducingStateDescriptor<>("sum", (a, b) -> a + b, Long.class));
}
public Tuple2<MyType, Long> map(MyType value) {
state.add(value.count());
return new Tuple2<>(value, state.get());
}
});
getReducingState
in interface RuntimeContext
T
- The type of value stored in the state.stateProperties
- The descriptor defining the properties of the stats.@PublicEvolving public <IN,ACC,OUT> AggregatingState<IN,OUT> getAggregatingState(AggregatingStateDescriptor<IN,ACC,OUT> stateProperties)
RuntimeContext
RuntimeContext.getState(ValueStateDescriptor)
, but is optimized for state that
aggregates values with different types.
This state is only accessible if the function is executed on a KeyedStream.
DataStream<MyType> stream = ...;
KeyedStream<MyType> keyedStream = stream.keyBy("id");
AggregateFunction<...> aggregateFunction = ...
keyedStream.map(new RichMapFunction<MyType, List<MyType>>() {
private AggregatingState<MyType, Long> state;
public void open(Configuration cfg) {
state = getRuntimeContext().getAggregatingState(
new AggregatingStateDescriptor<>("sum", aggregateFunction, Long.class));
}
public Tuple2<MyType, Long> map(MyType value) {
state.add(value);
return new Tuple2<>(value, state.get());
}
});
getAggregatingState
in interface RuntimeContext
IN
- The type of the values that are added to the state.ACC
- The type of the accumulator (intermediate aggregation state).OUT
- The type of the values that are returned from the state.stateProperties
- The descriptor defining the properties of the stats.@PublicEvolving @Deprecated public <T,ACC> FoldingState<T,ACC> getFoldingState(FoldingStateDescriptor<T,ACC> stateProperties)
RuntimeContext
RuntimeContext.getState(ValueStateDescriptor)
, but is optimized for state that
aggregates values with different types.
This state is only accessible if the function is executed on a KeyedStream.
DataStream<MyType> stream = ...;
KeyedStream<MyType> keyedStream = stream.keyBy("id");
keyedStream.map(new RichMapFunction<MyType, List<MyType>>() {
private FoldingState<MyType, Long> state;
public void open(Configuration cfg) {
state = getRuntimeContext().getFoldingState(
new FoldingStateDescriptor<>("sum", 0L, (a, b) -> a.count() + b, Long.class));
}
public Tuple2<MyType, Long> map(MyType value) {
state.add(value);
return new Tuple2<>(value, state.get());
}
});
getFoldingState
in interface RuntimeContext
T
- Type of the values folded in the other stateACC
- Type of the value in the statestateProperties
- The descriptor defining the properties of the stats.@PublicEvolving public <UK,UV> MapState<UK,UV> getMapState(MapStateDescriptor<UK,UV> stateProperties)
RuntimeContext
RuntimeContext.getState(ValueStateDescriptor)
, but is optimized for state that
is composed of user-defined key-value pairs
This state is only accessible if the function is executed on a KeyedStream.
DataStream<MyType> stream = ...;
KeyedStream<MyType> keyedStream = stream.keyBy("id");
keyedStream.map(new RichMapFunction<MyType, List<MyType>>() {
private MapState<MyType, Long> state;
public void open(Configuration cfg) {
state = getRuntimeContext().getMapState(
new MapStateDescriptor<>("sum", MyType.class, Long.class));
}
public Tuple2<MyType, Long> map(MyType value) {
return new Tuple2<>(value, state.get(value));
}
});
getMapState
in interface RuntimeContext
UK
- The type of the user keys stored in the state.UV
- The type of the user values stored in the state.stateProperties
- The descriptor defining the properties of the stats.Copyright © 2014–2018 The Apache Software Foundation. All rights reserved.