org.apache.flink.streaming.api.transformations

Class StreamTransformation<T>

java.lang.Object

org.apache.flink.streaming.api.transformations.StreamTransformation<T>

Type Parameters:

T - The type of the elements that result from this StreamTransformation

Direct Known Subclasses:

CoFeedbackTransformation, FeedbackTransformation, OneInputTransformation, PartitionTransformation, SelectTransformation, SinkTransformation, SourceTransformation, SplitTransformation, TwoInputTransformation, UnionTransformation

@Internal
public abstract class StreamTransformation<T>
extends Object

A StreamTransformation represents the operation that creates a DataStream. Every DataStream has an underlying StreamTransformation that is the origin of said DataStream.

API operations such as DataStream.map(org.apache.flink.api.common.functions.MapFunction<T, R>) create a tree of StreamTransformations underneath. When the stream program is to be executed this graph is translated to a StreamGraph using StreamGraphGenerator.

A StreamTransformation does not necessarily correspond to a physical operation at runtime. Some operations are only logical concepts. Examples of this are union, split/select data stream, partitioning.

The following graph of StreamTransformations:

   Source              Source        
      +                   +           
      |                   |           
      v                   v           
  Rebalance          HashPartition    
      +                   +           
      |                   |           
      |                   |           
      +------>Union<------+           
                +                     
                |                     
                v                     
              Split                   
                +                     
                |                     
                v                     
              Select                  
                +                     
                v                     
               Map                    
                +                     
                |                     
                v                     
              Sink 
 

Would result in this graph of operations at runtime:

  Source              Source
    +                   +
    |                   |
    |                   |
    +------->Map<-------+
              +
              |
              v
             Sink
 

The information about partitioning, union, split/select end up being encoded in the edges that connect the sources to the map operation.

Field Summary

Fields

Modifier and Type	Field and Description
protected long	bufferTimeout
protected int	id
protected static Integer	idCounter
protected String	name
protected TypeInformation<T>	outputType
protected boolean	typeUsed

Constructor Summary

Constructors

Constructor and Description
StreamTransformation(String name, TypeInformation<T> outputType, int parallelism) Creates a new StreamTransformation with the given name, output type and parallelism.

Method Summary

Modifier and Type	Method and Description
boolean	equals(Object o)
long	getBufferTimeout() Returns the buffer timeout of this StreamTransformation.
int	getId() Returns the unique ID of this StreamTransformation.
String	getName() Returns the name of this StreamTransformation.
static int	getNewNodeId()
TypeInformation<T>	getOutputType() Returns the output type of this StreamTransformation as a TypeInformation.
int	getParallelism() Returns the parallelism of this StreamTransformation
String	getSlotSharingGroup() Returns the slot sharing group of this transformation.
abstract Collection<StreamTransformation<?>>	getTransitivePredecessors() Returns all transitive predecessor StreamTransformations of this StreamTransformation.
String	getUid() Returns the user-specified ID of this transformation.
int	hashCode()
void	setBufferTimeout(long bufferTimeout) Set the buffer timeout of this StreamTransformation.
abstract void	setChainingStrategy(ChainingStrategy strategy) Sets the chaining strategy of this StreamTransformation.
void	setName(String name) Changes the name of this StreamTransformation.
void	setOutputType(TypeInformation<T> outputType) Tries to fill in the type information.
void	setParallelism(int parallelism) Sets the parallelism of this StreamTransformation
void	setSlotSharingGroup(String slotSharingGroup) Sets the slot sharing group of this transformation.
void	setUid(String uid) Sets an ID for this StreamTransformation.
String	toString()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

idCounter

protected static Integer idCounter

id

protected final int id

name

protected String name

outputType

protected TypeInformation<T> outputType

typeUsed

protected boolean typeUsed

bufferTimeout

protected long bufferTimeout

Constructor Detail

StreamTransformation

public StreamTransformation(String name,
                            TypeInformation<T> outputType,
                            int parallelism)

Creates a new StreamTransformation with the given name, output type and parallelism.

Parameters:

name - The name of the StreamTransformation, this will be shown in Visualizations and the Log

outputType - The output type of this StreamTransformation

parallelism - The parallelism of this StreamTransformation

Method Detail

getNewNodeId

public static int getNewNodeId()

getId

public int getId()

Returns the unique ID of this StreamTransformation.

setName

public void setName(String name)

Changes the name of this StreamTransformation.

getName

public String getName()

Returns the name of this StreamTransformation.

getParallelism

public int getParallelism()

Returns the parallelism of this StreamTransformation

setParallelism

public void setParallelism(int parallelism)

Sets the parallelism of this StreamTransformation

Parameters:

parallelism - The new parallelism to set on this StreamTransformation

setUid

public void setUid(String uid)

Sets an ID for this StreamTransformation.

The specified ID is used to assign the same operator ID across job submissions (for example when starting a job from a savepoint).

Important: this ID needs to be unique per transformation and job. Otherwise, job submission will fail.

Parameters:

uid - The unique user-specified ID of this transformation.

getUid

public String getUid()

Returns the user-specified ID of this transformation.

Returns:

The unique user-specified ID of this transformation.

getSlotSharingGroup

public String getSlotSharingGroup()

Returns the slot sharing group of this transformation.

See Also:

setSlotSharingGroup(String)

setSlotSharingGroup

public void setSlotSharingGroup(String slotSharingGroup)

Sets the slot sharing group of this transformation. Parallel instances of operations that are in the same slot sharing group will be co-located in the same TaskManager slot, if possible.

Initially, an operation is in the default slot sharing group. This can be explicitly set using setSlotSharingGroup("default").

Parameters:

slotSharingGroup - The slot sharing group name.

setOutputType

public void setOutputType(TypeInformation<T> outputType)

Tries to fill in the type information. Type information can be filled in later when the program uses a type hint. This method checks whether the type information has ever been accessed before and does not allow modifications if the type was accessed already. This ensures consistency by making sure different parts of the operation do not assume different type information.

Parameters:

outputType - The type information to fill in.

Throws:

IllegalStateException - Thrown, if the type information has been accessed before.

getOutputType

public TypeInformation<T> getOutputType()

Returns the output type of this StreamTransformation as a TypeInformation. Once this is used once the output type cannot be changed anymore using setOutputType(org.apache.flink.api.common.typeinfo.TypeInformation<T>).

Returns:

The output type of this StreamTransformation

setChainingStrategy

public abstract void setChainingStrategy(ChainingStrategy strategy)

Sets the chaining strategy of this StreamTransformation.

setBufferTimeout

public void setBufferTimeout(long bufferTimeout)

Set the buffer timeout of this StreamTransformation. The timeout is used when sending elements over the network. The timeout specifies how long a network buffer should be kept waiting before sending. A higher timeout means that more elements will be sent in one buffer, this increases throughput. The latency, however, is negatively affected by a higher timeout.

getBufferTimeout

public long getBufferTimeout()

Returns the buffer timeout of this StreamTransformation.

See Also:

setBufferTimeout(long)

getTransitivePredecessors

public abstract Collection<StreamTransformation<?>> getTransitivePredecessors()

Returns all transitive predecessor StreamTransformations of this StreamTransformation. This is, for example, used when determining whether a feedback edge of an iteration actually has the iteration head as a predecessor.

Returns:

The list of transitive predecessors.

toString

public String toString()

Overrides:

toString in class Object

equals

public boolean equals(Object o)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object