org.apache.flink.api.common.operators.base

Class DeltaIterationBase<ST,WT>

java.lang.Object

org.apache.flink.api.common.operators.Operator<OUT>

org.apache.flink.api.common.operators.AbstractUdfOperator<OUT,FT>

org.apache.flink.api.common.operators.DualInputOperator<ST,WT,ST,AbstractRichFunction>

org.apache.flink.api.common.operators.base.DeltaIterationBase<ST,WT>

All Implemented Interfaces:

IterationOperator, Visitable<Operator<?>>

@Internal
public class DeltaIterationBase<ST,WT>
extends DualInputOperator<ST,WT,ST,AbstractRichFunction>
implements IterationOperator

A DeltaIteration is similar to a BulkIterationBase, but maintains state across the individual iteration steps. The state is called the solution set, can be obtained via getSolutionSet(), and be accessed by joining (or CoGrouping) with it. The solution set is updated by producing a delta for it, which is merged into the solution set at the end of each iteration step.

The delta iteration must be closed by setting a delta for the solution set (setSolutionSetDelta(org.apache.flink.api.common.operators.Operator)) and the new workset (the data set that will be fed back, setNextWorkset(org.apache.flink.api.common.operators.Operator)). The DeltaIteration itself represents the result after the iteration has terminated. Delta iterations terminate when the feed back data set (the workset) is empty. In addition, a maximum number of steps is given as a fall back termination guard.

Elements in the solution set are uniquely identified by a key. When merging the solution set delta, contained elements with the same key are replaced.

This class is a subclass of DualInputOperator. The solution set is considered the first input, the workset is considered the second input.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DeltaIterationBase.SolutionSetPlaceHolder<ST> Specialized operator to use as a recognizable place-holder for the solution set input to the step function.
static class	DeltaIterationBase.WorksetPlaceHolder<WT> Specialized operator to use as a recognizable place-holder for the working set input to the step function.

Field Summary

Fields inherited from class org.apache.flink.api.common.operators.DualInputOperator

input1, input2

Fields inherited from class org.apache.flink.api.common.operators.AbstractUdfOperator

broadcastInputs, userFunction

Fields inherited from class org.apache.flink.api.common.operators.Operator

compilerHints, name, operatorInfo, parameters

Constructor Summary

Constructors

Constructor and Description
DeltaIterationBase(BinaryOperatorInformation<ST,WT,ST> operatorInfo, int keyPosition)
DeltaIterationBase(BinaryOperatorInformation<ST,WT,ST> operatorInfo, int[] keyPositions)
DeltaIterationBase(BinaryOperatorInformation<ST,WT,ST> operatorInfo, int[] keyPositions, String name)
DeltaIterationBase(BinaryOperatorInformation<ST,WT,ST> operatorInfo, int keyPosition, String name)

Method Summary

Modifier and Type	Method and Description
protected List<ST>	executeOnCollections(List<ST> inputData1, List<WT> inputData2, RuntimeContext runtimeContext, ExecutionConfig executionConfig)
AggregatorRegistry	getAggregators()
Map<String,Operator<?>>	getBroadcastInputs() DeltaIteration meta operator cannot have broadcast inputs.
Operator<ST>	getInitialSolutionSet() Returns the initial solution set input, or null, if none is set.
Operator<WT>	getInitialWorkset() Returns the initial workset input, or null, if none is set.
int	getMaximumNumberOfIterations()
Operator<WT>	getNextWorkset() Gets the contract that has been set as the next workset.
Operator<ST>	getSolutionSet() Gets the contract that represents the solution set for the step function.
Operator<ST>	getSolutionSetDelta() Gets the contract that has been set as the solution set delta.
int[]	getSolutionSetKeyFields()
Operator<WT>	getWorkset() Gets the contract that represents the workset for the step function.
boolean	isSolutionSetUnManaged() gets whether the solution set is in managed or unmanaged memory.
void	setBroadcastVariable(String name, Operator<?> root) The DeltaIteration meta operator cannot have broadcast inputs.
<X> void	setBroadcastVariables(Map<String,Operator<X>> inputs) The DeltaIteration meta operator cannot have broadcast inputs.
void	setInitialSolutionSet(Operator<ST> input) Sets the given input as the initial solution set.
void	setInitialWorkset(Operator<WT> input) Sets the given input as the initial workset.
void	setMaximumNumberOfIterations(int maxIterations)
void	setNextWorkset(Operator<WT> result) Sets the contract of the step function that represents the next workset.
void	setSolutionSetDelta(Operator<ST> delta) Sets the contract of the step function that represents the solution set delta.
void	setSolutionSetUnManaged(boolean solutionSetUnManaged) Sets whether to keep the solution set in managed memory (safe against heap exhaustion) or unmanaged memory (objects on heap).

Methods inherited from class org.apache.flink.api.common.operators.DualInputOperator

accept, addFirstInput, addFirstInputs, addSecondInput, addSecondInputs, clearFirstInput, clearSecondInput, getFirstInput, getKeyColumns, getNumberOfInputs, getOperatorInfo, getSecondInput, getSemanticProperties, setFirstInput, setFirstInput, setFirstInputs, setSecondInput, setSecondInput, setSecondInputs, setSemanticProperties

Methods inherited from class org.apache.flink.api.common.operators.AbstractUdfOperator

asArray, emptyClassArray, getUserCodeWrapper

Methods inherited from class org.apache.flink.api.common.operators.Operator

createUnionCascade, createUnionCascade, createUnionCascade, getCompilerHints, getMinResources, getName, getParallelism, getParameters, getPreferredResources, setName, setParallelism, setParameter, setParameter, setParameter, setResources, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

DeltaIterationBase

public DeltaIterationBase(BinaryOperatorInformation<ST,WT,ST> operatorInfo,
                          int keyPosition)

DeltaIterationBase

public DeltaIterationBase(BinaryOperatorInformation<ST,WT,ST> operatorInfo,
                          int[] keyPositions)

DeltaIterationBase

public DeltaIterationBase(BinaryOperatorInformation<ST,WT,ST> operatorInfo,
                          int keyPosition,
                          String name)

DeltaIterationBase

public DeltaIterationBase(BinaryOperatorInformation<ST,WT,ST> operatorInfo,
                          int[] keyPositions,
                          String name)

Method Detail

getSolutionSetKeyFields

public int[] getSolutionSetKeyFields()

setMaximumNumberOfIterations

public void setMaximumNumberOfIterations(int maxIterations)

getMaximumNumberOfIterations

public int getMaximumNumberOfIterations()

getAggregators

public AggregatorRegistry getAggregators()

Specified by:

getAggregators in interface IterationOperator

getSolutionSet

public Operator<ST> getSolutionSet()

Gets the contract that represents the solution set for the step function.

Returns:

The solution set for the step function.

getWorkset

public Operator<WT> getWorkset()

Gets the contract that represents the workset for the step function.

Returns:

The workset for the step function.

setNextWorkset

public void setNextWorkset(Operator<WT> result)

Sets the contract of the step function that represents the next workset. This contract is considered one of the two sinks of the step function (the other one being the solution set delta).

Parameters:

result - The contract representing the next workset.

getNextWorkset

public Operator<WT> getNextWorkset()

Gets the contract that has been set as the next workset.

Returns:

The contract that has been set as the next workset.

setSolutionSetDelta

public void setSolutionSetDelta(Operator<ST> delta)

Sets the contract of the step function that represents the solution set delta. This contract is considered one of the two sinks of the step function (the other one being the next workset).

Parameters:

delta - The contract representing the solution set delta.

getSolutionSetDelta

public Operator<ST> getSolutionSetDelta()

Gets the contract that has been set as the solution set delta.

Returns:

The contract that has been set as the solution set delta.

getInitialSolutionSet

public Operator<ST> getInitialSolutionSet()

Returns the initial solution set input, or null, if none is set.

Returns:

The iteration's initial solution set input.

getInitialWorkset

public Operator<WT> getInitialWorkset()

Returns the initial workset input, or null, if none is set.

Returns:

The iteration's workset input.

setInitialSolutionSet

public void setInitialSolutionSet(Operator<ST> input)

Sets the given input as the initial solution set.

Parameters:

input - The contract to set the initial solution set.

setInitialWorkset

public void setInitialWorkset(Operator<WT> input)

Sets the given input as the initial workset.

Parameters:

input - The contract to set as the initial workset.

getBroadcastInputs

public Map<String,Operator<?>> getBroadcastInputs()

DeltaIteration meta operator cannot have broadcast inputs.

Overrides:

getBroadcastInputs in class AbstractUdfOperator<ST,AbstractRichFunction>

Returns:

An empty map.

setBroadcastVariable

public void setBroadcastVariable(String name,
                                 Operator<?> root)

The DeltaIteration meta operator cannot have broadcast inputs. This method always throws an exception.

Overrides:

setBroadcastVariable in class AbstractUdfOperator<ST,AbstractRichFunction>

Parameters:

name - Ignored.

root - Ignored.

setBroadcastVariables

public <X> void setBroadcastVariables(Map<String,Operator<X>> inputs)

The DeltaIteration meta operator cannot have broadcast inputs. This method always throws an exception.

Overrides:

setBroadcastVariables in class AbstractUdfOperator<ST,AbstractRichFunction>

Parameters:

inputs - Ignored

setSolutionSetUnManaged

public void setSolutionSetUnManaged(boolean solutionSetUnManaged)

Sets whether to keep the solution set in managed memory (safe against heap exhaustion) or unmanaged memory (objects on heap).

Parameters:

solutionSetUnManaged - True to keep the solution set in unmanaged memory, false to keep it in managed memory.

See Also:

isSolutionSetUnManaged()

isSolutionSetUnManaged

public boolean isSolutionSetUnManaged()

gets whether the solution set is in managed or unmanaged memory.

Returns:

True, if the solution set is in unmanaged memory (object heap), false if in managed memory.

See Also:

setSolutionSetUnManaged(boolean)

executeOnCollections

protected List<ST> executeOnCollections(List<ST> inputData1,
                                        List<WT> inputData2,
                                        RuntimeContext runtimeContext,
                                        ExecutionConfig executionConfig)

Specified by:

executeOnCollections in class DualInputOperator<ST,WT,ST,AbstractRichFunction>