org.apache.flink.streaming.api.graph

Class StreamGraphGenerator

java.lang.Object

org.apache.flink.streaming.api.graph.StreamGraphGenerator

@Internal
public class StreamGraphGenerator
extends Object

A generator that generates a StreamGraph from a graph of StreamTransformations.

This traverses the tree of StreamTransformations starting from the sinks. At each transformation we recursively transform the inputs, then create a node in the StreamGraph and add edges from the input Nodes to our newly created node. The transformation methods return the IDs of the nodes in the StreamGraph that represent the input transformation. Several IDs can be returned to be able to deal with feedback transformations and unions.

Partitioning, split/select and union don't create actual nodes in the StreamGraph. For these, we create a virtual node in the StreamGraph that holds the specific property, i.e. partitioning, selector and so on. When an edge is created from a virtual node to a downstream node the StreamGraph resolved the id of the original node and creates an edge in the graph with the desired property. For example, if you have this graph:

     Map-1 -> HashPartition-2 -> Map-3
 

where the numbers represent transformation IDs. We first recurse all the way down. Map-1 is transformed, i.e. we create a StreamNode with ID 1. Then we transform the HashPartition, for this, we create virtual node of ID 4 that holds the property HashPartition. This transformation returns the ID 4. Then we transform the Map-3. We add the edge 4 -> 3. The StreamGraph resolved the actual node with ID 1 and creates and edge 1 -> 3 with the property HashPartition.

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_LOWER_BOUND_MAX_PARALLELISM
protected static Integer	iterationIdCounter
static int	UPPER_BOUND_MAX_PARALLELISM

Method Summary

Modifier and Type	Method and Description
static StreamGraph	generate(StreamExecutionEnvironment env, List<StreamTransformation<?>> transformations) Generates a StreamGraph by traversing the graph of StreamTransformations starting from the given transformations.
static int	getNewIterationNodeId()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_LOWER_BOUND_MAX_PARALLELISM

public static final int DEFAULT_LOWER_BOUND_MAX_PARALLELISM

See Also:

Constant Field Values

UPPER_BOUND_MAX_PARALLELISM

public static final int UPPER_BOUND_MAX_PARALLELISM

See Also:

Constant Field Values

iterationIdCounter

protected static Integer iterationIdCounter

Method Detail

getNewIterationNodeId

public static int getNewIterationNodeId()

generate

public static StreamGraph generate(StreamExecutionEnvironment env,
                                   List<StreamTransformation<?>> transformations)

Generates a StreamGraph by traversing the graph of StreamTransformations starting from the given transformations.

Parameters:

env - The StreamExecutionEnvironment that is used to set some parameters of the job

transformations - The transformations starting from which to transform the graph

Returns:

The generated StreamGraph