@Experimental public class Iterations extends Object
The iteration body will be invoked with two parameters: The first parameter is a list of input variable streams, which are created as the union of the initial variable streams and the corresponding feedback variable streams (returned by the iteration body); The second parameter is the data streams given to this method.
During the execution of iteration body, each of the records involved in the iteration has an epoch attached, which is mark the progress of the iteration. The epoch is computed as:
The framework would given the notification at the end of each epoch for operators and UDFs
that implements IterationListener
.
The limitation of constructing the subgraph inside the iteration body could be refer in IterationBody
.
An example of the iteration is like:
DataStreamList result = Iterations.iterateUnboundedStreams(
DataStreamList.of(first, second),
DataStreamList.of(third),
(variableStreams, dataStreams) -> {
...
return new IterationBodyResult(
DataStreamList.of(firstFeedback, secondFeedback),
DataStreamList.of(output));
}
result.<Integer>get(0).addSink(...);
Constructor and Description |
---|
Iterations() |
Modifier and Type | Method and Description |
---|---|
static DataStreamList |
iterateBoundedStreamsUntilTermination(DataStreamList initVariableStreams,
ReplayableDataStreamList dataStreams,
IterationConfig config,
IterationBody body)
This method uses an iteration body to process records in some bounded data streams
iteratively until no more records are iterating or the given terminating criteria stream is
empty in one round.
|
static DataStreamList |
iterateUnboundedStreams(DataStreamList initVariableStreams,
DataStreamList dataStreams,
IterationBody body)
This method uses an iteration body to process records in possibly unbounded data streams.
|
public static DataStreamList iterateUnboundedStreams(DataStreamList initVariableStreams, DataStreamList dataStreams, IterationBody body)
initVariableStreams
- The initial variable streams, which is merged with the feedback
variable streams before being used as the 1st parameter to invoke the iteration body.dataStreams
- The non-variable streams also refer in the body
.body
- The computation logic which takes variable/data streams and returns
feedback/output streams.public static DataStreamList iterateBoundedStreamsUntilTermination(DataStreamList initVariableStreams, ReplayableDataStreamList dataStreams, IterationConfig config, IterationBody body)
initVariableStreams
- The initial variable streams, which is merged with the feedback
variable streams before being used as the 1st parameter to invoke the iteration body.dataStreams
- The non-variable streams also refer in the body
and if each of
them needs replayed for each round.config
- The config for the iteration, like whether to re-create the operator on each
round.body
- The computation logic which takes variable/data streams and returns
feedback/output streams.Copyright © 2019–2023 The Apache Software Foundation. All rights reserved.