@Internal public class StreamGraphGenerator extends Object
StreamGraph
from a graph of
Transformation
s.
This traverses the tree of Transformations
starting from the sinks. At each
transformation we recursively transform the inputs, then create a node in the StreamGraph
and add edges from the input Nodes to our newly created node. The transformation methods
return the IDs of the nodes in the StreamGraph that represent the input transformation. Several
IDs can be returned to be able to deal with feedback transformations and unions.
Partitioning, split/select and union don't create actual nodes in the StreamGraph
. For
these, we create a virtual node in the StreamGraph
that holds the specific property, i.e.
partitioning, selector and so on. When an edge is created from a virtual node to a downstream
node the StreamGraph
resolved the id of the original node and creates an edge
in the graph with the desired property. For example, if you have this graph:
Map-1 -> HashPartition-2 -> Map-3
where the numbers represent transformation IDs. We first recurse all the way down. Map-1
is transformed, i.e. we create a StreamNode
with ID 1. Then we transform the
HashPartition
, for this, we create virtual node of ID 4 that holds the property
HashPartition
. This transformation returns the ID 4. Then we transform the Map-3
.
We add the edge 4 -> 3
. The StreamGraph
resolved the actual node with ID 1 and
creates and edge 1 -> 3
with the property HashPartition.
Modifier and Type | Field and Description |
---|---|
static String |
DEFAULT_JOB_NAME |
static int |
DEFAULT_LOWER_BOUND_MAX_PARALLELISM |
static ScheduleMode |
DEFAULT_SCHEDULE_MODE |
static String |
DEFAULT_SLOT_SHARING_GROUP |
static TimeCharacteristic |
DEFAULT_TIME_CHARACTERISTIC |
protected static Integer |
iterationIdCounter |
Constructor and Description |
---|
StreamGraphGenerator(List<Transformation<?>> transformations,
ExecutionConfig executionConfig,
CheckpointConfig checkpointConfig) |
Modifier and Type | Method and Description |
---|---|
StreamGraph |
generate() |
static int |
getNewIterationNodeId() |
StreamGraphGenerator |
setBlockingConnectionsBetweenChains(boolean blockingConnectionsBetweenChains) |
StreamGraphGenerator |
setChaining(boolean chaining) |
StreamGraphGenerator |
setDefaultBufferTimeout(long defaultBufferTimeout) |
StreamGraphGenerator |
setJobName(String jobName) |
void |
setSavepointRestoreSettings(SavepointRestoreSettings savepointRestoreSettings) |
StreamGraphGenerator |
setScheduleMode(ScheduleMode scheduleMode) |
StreamGraphGenerator |
setStateBackend(StateBackend stateBackend) |
StreamGraphGenerator |
setTimeCharacteristic(TimeCharacteristic timeCharacteristic) |
StreamGraphGenerator |
setUserArtifacts(Collection<Tuple2<String,DistributedCache.DistributedCacheEntry>> userArtifacts) |
public static final int DEFAULT_LOWER_BOUND_MAX_PARALLELISM
public static final ScheduleMode DEFAULT_SCHEDULE_MODE
public static final TimeCharacteristic DEFAULT_TIME_CHARACTERISTIC
public static final String DEFAULT_JOB_NAME
public static final String DEFAULT_SLOT_SHARING_GROUP
protected static Integer iterationIdCounter
public StreamGraphGenerator(List<Transformation<?>> transformations, ExecutionConfig executionConfig, CheckpointConfig checkpointConfig)
public static int getNewIterationNodeId()
public StreamGraphGenerator setStateBackend(StateBackend stateBackend)
public StreamGraphGenerator setChaining(boolean chaining)
public StreamGraphGenerator setScheduleMode(ScheduleMode scheduleMode)
public StreamGraphGenerator setUserArtifacts(Collection<Tuple2<String,DistributedCache.DistributedCacheEntry>> userArtifacts)
public StreamGraphGenerator setTimeCharacteristic(TimeCharacteristic timeCharacteristic)
public StreamGraphGenerator setDefaultBufferTimeout(long defaultBufferTimeout)
public StreamGraphGenerator setJobName(String jobName)
public StreamGraphGenerator setBlockingConnectionsBetweenChains(boolean blockingConnectionsBetweenChains)
public void setSavepointRestoreSettings(SavepointRestoreSettings savepointRestoreSettings)
public StreamGraph generate()
Copyright © 2014–2020 The Apache Software Foundation. All rights reserved.