org.apache.flink.streaming.api.graph

Class StreamGraphGenerator

java.lang.Object

org.apache.flink.streaming.api.graph.StreamGraphGenerator

@Internal
public class StreamGraphGenerator
extends Object

A generator that generates a StreamGraph from a graph of Transformations.

This traverses the tree of Transformations starting from the sinks. At each transformation we recursively transform the inputs, then create a node in the StreamGraph and add edges from the input Nodes to our newly created node. The transformation methods return the IDs of the nodes in the StreamGraph that represent the input transformation. Several IDs can be returned to be able to deal with feedback transformations and unions.

Partitioning, split/select and union don't create actual nodes in the StreamGraph. For these, we create a virtual node in the StreamGraph that holds the specific property, i.e. partitioning, selector and so on. When an edge is created from a virtual node to a downstream node the StreamGraph resolved the id of the original node and creates an edge in the graph with the desired property. For example, if you have this graph:

     Map-1 -> HashPartition-2 -> Map-3
 

where the numbers represent transformation IDs. We first recurse all the way down. Map-1 is transformed, i.e. we create a StreamNode with ID 1. Then we transform the HashPartition, for this, we create virtual node of ID 4 that holds the property HashPartition. This transformation returns the ID 4. Then we transform the Map-3. We add the edge 4 -> 3. The StreamGraph resolved the actual node with ID 1 and creates and edge 1 -> 3 with the property HashPartition.

Field Summary

Fields

Modifier and Type	Field and Description
static String	DEFAULT_BATCH_JOB_NAME
static int	DEFAULT_LOWER_BOUND_MAX_PARALLELISM
static String	DEFAULT_SLOT_SHARING_GROUP
static String	DEFAULT_STREAMING_JOB_NAME
static TimeCharacteristic	DEFAULT_TIME_CHARACTERISTIC
protected static Integer	iterationIdCounter

Constructor Summary

Constructors

Constructor and Description
StreamGraphGenerator(List<Transformation<?>> transformations, ExecutionConfig executionConfig, CheckpointConfig checkpointConfig)
StreamGraphGenerator(List<Transformation<?>> transformations, ExecutionConfig executionConfig, CheckpointConfig checkpointConfig, Configuration configuration)

Method Summary

Modifier and Type	Method and Description
StreamGraph	generate()
static int	getNewIterationNodeId()
void	setSavepointRestoreSettings(SavepointRestoreSettings savepointRestoreSettings)
StreamGraphGenerator	setSlotSharingGroupResource(Map<String,ResourceProfile> slotSharingGroupResources) Specify fine-grained resource requirements for slot sharing groups.
StreamGraphGenerator	setStateBackend(StateBackend stateBackend)
StreamGraphGenerator	setTimeCharacteristic(TimeCharacteristic timeCharacteristic)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_LOWER_BOUND_MAX_PARALLELISM

public static final int DEFAULT_LOWER_BOUND_MAX_PARALLELISM

See Also:

Constant Field Values

DEFAULT_TIME_CHARACTERISTIC

public static final TimeCharacteristic DEFAULT_TIME_CHARACTERISTIC

DEFAULT_STREAMING_JOB_NAME

public static final String DEFAULT_STREAMING_JOB_NAME

See Also:

Constant Field Values

DEFAULT_BATCH_JOB_NAME

public static final String DEFAULT_BATCH_JOB_NAME

See Also:

Constant Field Values

DEFAULT_SLOT_SHARING_GROUP

public static final String DEFAULT_SLOT_SHARING_GROUP

See Also:

Constant Field Values

iterationIdCounter

protected static Integer iterationIdCounter

Constructor Detail

StreamGraphGenerator

public StreamGraphGenerator(List<Transformation<?>> transformations,
                            ExecutionConfig executionConfig,
                            CheckpointConfig checkpointConfig)

StreamGraphGenerator

public StreamGraphGenerator(List<Transformation<?>> transformations,
                            ExecutionConfig executionConfig,
                            CheckpointConfig checkpointConfig,
                            Configuration configuration)

Method Detail

getNewIterationNodeId

public static int getNewIterationNodeId()

setStateBackend

public StreamGraphGenerator setStateBackend(StateBackend stateBackend)

setTimeCharacteristic

public StreamGraphGenerator setTimeCharacteristic(TimeCharacteristic timeCharacteristic)

setSlotSharingGroupResource

public StreamGraphGenerator setSlotSharingGroupResource(Map<String,ResourceProfile> slotSharingGroupResources)

Specify fine-grained resource requirements for slot sharing groups.

Note that a slot sharing group hints the scheduler that the grouped operators CAN be deployed into a shared slot. There's no guarantee that the scheduler always deploy the grouped operators together. In cases grouped operators are deployed into separate slots, the slot resources will be derived from the specified group requirements.

setSavepointRestoreSettings

public void setSavepointRestoreSettings(SavepointRestoreSettings savepointRestoreSettings)

generate

public StreamGraph generate()