org.apache.flink.streaming.api.datastream

Class DataStream<T>

java.lang.Object

org.apache.flink.streaming.api.datastream.DataStream<T>

Type Parameters:

T - The type of the elements in this stream.

Direct Known Subclasses:

CachedDataStream, KeyedStream, SideOutputDataStream, SingleOutputStreamOperator

@Public
public class DataStream<T>
extends Object

A DataStream represents a stream of elements of the same type. A DataStream can be transformed into another DataStream by applying a transformation as for example:

• map(org.apache.flink.api.common.functions.MapFunction<T, R>)
• filter(org.apache.flink.api.common.functions.FilterFunction<T>)

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DataStream.Collector<T> This class acts as an accessor to elements collected via collectAsync(Collector).

Field Summary

Fields

Modifier and Type	Field and Description
protected StreamExecutionEnvironment	environment
protected Transformation<T>	transformation

Constructor Summary

Constructors

Constructor and Description
DataStream(StreamExecutionEnvironment environment, Transformation<T> transformation) Create a new DataStream in the given execution environment with partitioning set to forward by default.

Method Summary

Modifier and Type	Method and Description
DataStreamSink<T>	addSink(SinkFunction<T> sinkFunction) Adds the given sink to this DataStream.
SingleOutputStreamOperator<T>	assignTimestampsAndWatermarks(AssignerWithPeriodicWatermarks<T> timestampAndWatermarkAssigner) Deprecated. Please use assignTimestampsAndWatermarks(WatermarkStrategy) instead.
SingleOutputStreamOperator<T>	assignTimestampsAndWatermarks(AssignerWithPunctuatedWatermarks<T> timestampAndWatermarkAssigner) Deprecated. Please use assignTimestampsAndWatermarks(WatermarkStrategy) instead.
SingleOutputStreamOperator<T>	assignTimestampsAndWatermarks(WatermarkStrategy<T> watermarkStrategy) Assigns timestamps to the elements in the data stream and generates watermarks to signal event time progress.
DataStream<T>	broadcast() Sets the partitioning of the DataStream so that the output elements are broadcasted to every parallel instance of the next operation.
BroadcastStream<T>	broadcast(MapStateDescriptor<?,?>... broadcastStateDescriptors) Sets the partitioning of the DataStream so that the output elements are broadcasted to every parallel instance of the next operation.
protected <F> F	clean(F f) Invokes the ClosureCleaner on the given function if closure cleaning is enabled in the ExecutionConfig.
<T2> CoGroupedStreams<T,T2>	coGroup(DataStream<T2> otherStream) Creates a join operation.
CloseableIterator<T>	collectAsync() Sets up the collection of the elements in this DataStream, and returns an iterator over the collected elements that can be used to retrieve elements once the job execution has started.
void	collectAsync(DataStream.Collector<T> collector) Sets up the collection of the elements in this DataStream, which can be retrieved later via the given DataStream.Collector.
<R> BroadcastConnectedStream<T,R>	connect(BroadcastStream<R> broadcastStream) Creates a new BroadcastConnectedStream by connecting the current DataStream or KeyedStream with a BroadcastStream.
<R> ConnectedStreams<T,R>	connect(DataStream<R> dataStream) Creates a new ConnectedStreams by connecting DataStream outputs of (possible) different types with each other.
AllWindowedStream<T,GlobalWindow>	countWindowAll(long size) Windows this DataStream into tumbling count windows.
AllWindowedStream<T,GlobalWindow>	countWindowAll(long size, long slide) Windows this DataStream into sliding count windows.
protected <R> SingleOutputStreamOperator<R>	doTransform(String operatorName, TypeInformation<R> outTypeInfo, StreamOperatorFactory<R> operatorFactory)
CloseableIterator<T>	executeAndCollect() Triggers the distributed execution of the streaming dataflow and returns an iterator over the elements of the given DataStream.
List<T>	executeAndCollect(int limit) Triggers the distributed execution of the streaming dataflow and returns an iterator over the elements of the given DataStream.
CloseableIterator<T>	executeAndCollect(String jobExecutionName) Triggers the distributed execution of the streaming dataflow and returns an iterator over the elements of the given DataStream.
List<T>	executeAndCollect(String jobExecutionName, int limit) Triggers the distributed execution of the streaming dataflow and returns an iterator over the elements of the given DataStream.
SingleOutputStreamOperator<T>	filter(FilterFunction<T> filter) Applies a Filter transformation on a DataStream.
<R> SingleOutputStreamOperator<R>	flatMap(FlatMapFunction<T,R> flatMapper) Applies a FlatMap transformation on a DataStream.
<R> SingleOutputStreamOperator<R>	flatMap(FlatMapFunction<T,R> flatMapper, TypeInformation<R> outputType) Applies a FlatMap transformation on a DataStream.
DataStream<T>	forward() Sets the partitioning of the DataStream so that the output elements are forwarded to the local subtask of the next operation.
PartitionWindowedStream<T>	fullWindowPartition() Collect records from each partition into a separate full window.
ExecutionConfig	getExecutionConfig()
StreamExecutionEnvironment	getExecutionEnvironment() Returns the StreamExecutionEnvironment that was used to create this DataStream.
int	getId() Returns the ID of the DataStream in the current StreamExecutionEnvironment.
ResourceSpec	getMinResources() Gets the minimum resources for this operator.
int	getParallelism() Gets the parallelism for this operator.
ResourceSpec	getPreferredResources() Gets the preferred resources for this operator.
Transformation<T>	getTransformation() Returns the Transformation that represents the operation that logically creates this DataStream.
TypeInformation<T>	getType() Gets the type of the stream.
DataStream<T>	global() Sets the partitioning of the DataStream so that the output values all go to the first instance of the next processing operator.
IterativeStream<T>	iterate() Deprecated. This method is deprecated since Flink 1.19. The only known use case of this Iteration API comes from Flink ML, which already has its own implementation of iteration and no longer uses this API. If there's any use cases other than Flink ML that needs iteration support, please reach out to dev@flink.apache.org and we can consider making the Flink ML iteration implementation a separate common library.
IterativeStream<T>	iterate(long maxWaitTimeMillis) Deprecated. This method is deprecated since Flink 1.19. The only known use case of this Iteration API comes from Flink ML, which already has its own implementation of iteration and no longer uses this API. If there's any use cases other than Flink ML that needs iteration support, please reach out to dev@flink.apache.org and we can consider making the Flink ML iteration implementation a separate common library.
<T2> JoinedStreams<T,T2>	join(DataStream<T2> otherStream) Creates a join operation.
KeyedStream<T,Tuple>	keyBy(int... fields) Deprecated. Use keyBy(KeySelector).
<K> KeyedStream<T,K>	keyBy(KeySelector<T,K> key) It creates a new KeyedStream that uses the provided key for partitioning its operator states.
<K> KeyedStream<T,K>	keyBy(KeySelector<T,K> key, TypeInformation<K> keyType) It creates a new KeyedStream that uses the provided key with explicit type information for partitioning its operator states.
KeyedStream<T,Tuple>	keyBy(String... fields) Deprecated. Use keyBy(KeySelector).
<R> SingleOutputStreamOperator<R>	map(MapFunction<T,R> mapper) Applies a Map transformation on a DataStream.
<R> SingleOutputStreamOperator<R>	map(MapFunction<T,R> mapper, TypeInformation<R> outputType) Applies a Map transformation on a DataStream.
<K> DataStream<T>	partitionCustom(Partitioner<K> partitioner, int field) Deprecated. use partitionCustom(Partitioner, KeySelector).
<K> DataStream<T>	partitionCustom(Partitioner<K> partitioner, KeySelector<T,K> keySelector) Partitions a DataStream on the key returned by the selector, using a custom partitioner.
<K> DataStream<T>	partitionCustom(Partitioner<K> partitioner, String field) Deprecated. use partitionCustom(Partitioner, KeySelector).
DataStreamSink<T>	print() Writes a DataStream to the standard output stream (stdout).
DataStreamSink<T>	print(String sinkIdentifier) Writes a DataStream to the standard output stream (stdout).
DataStreamSink<T>	printToErr() Writes a DataStream to the standard error stream (stderr).
DataStreamSink<T>	printToErr(String sinkIdentifier) Writes a DataStream to the standard error stream (stderr).
<R> SingleOutputStreamOperator<R>	process(ProcessFunction<T,R> processFunction) Applies the given ProcessFunction on the input stream, thereby creating a transformed output stream.
<R> SingleOutputStreamOperator<R>	process(ProcessFunction<T,R> processFunction, TypeInformation<R> outputType) Applies the given ProcessFunction on the input stream, thereby creating a transformed output stream.
<R extends Tuple> SingleOutputStreamOperator<R>	project(int... fieldIndexes) Initiates a Project transformation on a Tuple DataStream.
DataStream<T>	rebalance() Sets the partitioning of the DataStream so that the output elements are distributed evenly to instances of the next operation in a round-robin fashion.
DataStream<T>	rescale() Sets the partitioning of the DataStream so that the output elements are distributed evenly to a subset of instances of the next operation in a round-robin fashion.
protected DataStream<T>	setConnectionType(StreamPartitioner<T> partitioner) Internal function for setting the partitioner for the DataStream.
DataStream<T>	shuffle() Sets the partitioning of the DataStream so that the output elements are shuffled uniformly randomly to the next operation.
DataStreamSink<T>	sinkTo(Sink<T,?,?,?> sink) Adds the given Sink to this DataStream.
DataStreamSink<T>	sinkTo(Sink<T,?,?,?> sink, CustomSinkOperatorUidHashes customSinkOperatorUidHashes) Adds the given Sink to this DataStream.
DataStreamSink<T>	sinkTo(Sink<T> sink) Adds the given Sink to this DataStream.
DataStreamSink<T>	sinkTo(Sink<T> sink, CustomSinkOperatorUidHashes customSinkOperatorUidHashes) Adds the given Sink to this DataStream.
AllWindowedStream<T,TimeWindow>	timeWindowAll(Time size) Deprecated. Please use windowAll(WindowAssigner) with either TumblingEventTimeWindows or TumblingProcessingTimeWindows. For more information, see the deprecation notice on TimeCharacteristic
AllWindowedStream<T,TimeWindow>	timeWindowAll(Time size, Time slide) Deprecated. Please use windowAll(WindowAssigner) with either SlidingEventTimeWindows or SlidingProcessingTimeWindows. For more information, see the deprecation notice on TimeCharacteristic
<R> SingleOutputStreamOperator<R>	transform(String operatorName, TypeInformation<R> outTypeInfo, OneInputStreamOperator<T,R> operator) Method for passing user defined operators along with the type information that will transform the DataStream.
<R> SingleOutputStreamOperator<R>	transform(String operatorName, TypeInformation<R> outTypeInfo, OneInputStreamOperatorFactory<T,R> operatorFactory) Method for passing user defined operators created by the given factory along with the type information that will transform the DataStream.
DataStream<T>	union(DataStream<T>... streams) Creates a new DataStream by merging DataStream outputs of the same type with each other.
<W extends Window> AllWindowedStream<T,W>	windowAll(WindowAssigner<? super T,W> assigner) Windows this data stream to a AllWindowedStream, which evaluates windows over a non key grouped stream.
DataStreamSink<T>	writeAsCsv(String path) Deprecated. Please use the StreamingFileSink explicitly using the addSink(SinkFunction) method.
DataStreamSink<T>	writeAsCsv(String path, FileSystem.WriteMode writeMode) Deprecated. Please use the StreamingFileSink explicitly using the addSink(SinkFunction) method.
<X extends Tuple> DataStreamSink<T>	writeAsCsv(String path, FileSystem.WriteMode writeMode, String rowDelimiter, String fieldDelimiter) Deprecated. Please use the StreamingFileSink explicitly using the addSink(SinkFunction) method.
DataStreamSink<T>	writeAsText(String path) Deprecated. Please use the StreamingFileSink explicitly using the addSink(SinkFunction) method.
DataStreamSink<T>	writeAsText(String path, FileSystem.WriteMode writeMode) Deprecated. Please use the StreamingFileSink explicitly using the addSink(SinkFunction) method.
DataStreamSink<T>	writeToSocket(String hostName, int port, SerializationSchema<T> schema) Writes the DataStream to a socket as a byte array.
DataStreamSink<T>	writeUsingOutputFormat(OutputFormat<T> format) Deprecated. Please use the StreamingFileSink explicitly using the addSink(SinkFunction) method.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

environment

protected final StreamExecutionEnvironment environment

transformation

protected final Transformation<T> transformation

Constructor Detail

DataStream

public DataStream(StreamExecutionEnvironment environment,
                  Transformation<T> transformation)

Create a new DataStream in the given execution environment with partitioning set to forward by default.

Parameters:

environment - The StreamExecutionEnvironment

Method Detail

getId

@Internal
public int getId()

Returns the ID of the DataStream in the current StreamExecutionEnvironment.

Returns:

ID of the DataStream

getParallelism

public int getParallelism()

Gets the parallelism for this operator.

Returns:

The parallelism set for this operator.

getMinResources

@PublicEvolving
public ResourceSpec getMinResources()

Gets the minimum resources for this operator.

Returns:

The minimum resources set for this operator.

getPreferredResources

@PublicEvolving
public ResourceSpec getPreferredResources()

Gets the preferred resources for this operator.

Returns:

The preferred resources set for this operator.

getType

public TypeInformation<T> getType()

Gets the type of the stream.

Returns:

The type of the datastream.

clean

protected <F> F clean(F f)

Invokes the ClosureCleaner on the given function if closure cleaning is enabled in the ExecutionConfig.

Returns:

The cleaned Function

getExecutionEnvironment

public StreamExecutionEnvironment getExecutionEnvironment()

Returns the StreamExecutionEnvironment that was used to create this DataStream.

Returns:

The Execution Environment

getExecutionConfig

public ExecutionConfig getExecutionConfig()

union

@SafeVarargs
public final DataStream<T> union(DataStream<T>... streams)

Creates a new DataStream by merging DataStream outputs of the same type with each other. The DataStreams merged using this operator will be transformed simultaneously.

Parameters:

streams - The DataStreams to union output with.

Returns:

The DataStream.

connect

public <R> ConnectedStreams<T,R> connect(DataStream<R> dataStream)

Creates a new ConnectedStreams by connecting DataStream outputs of (possible) different types with each other. The DataStreams connected using this operator can be used with CoFunctions to apply joint transformations.

Parameters:

dataStream - The DataStream with which this stream will be connected.

Returns:

The ConnectedStreams.

connect

@PublicEvolving
public <R> BroadcastConnectedStream<T,R> connect(BroadcastStream<R> broadcastStream)

Creates a new BroadcastConnectedStream by connecting the current DataStream or KeyedStream with a BroadcastStream.

The latter can be created using the broadcast(MapStateDescriptor[]) method.

The resulting stream can be further processed using the BroadcastConnectedStream.process(MyFunction) method, where MyFunction can be either a KeyedBroadcastProcessFunction or a BroadcastProcessFunction depending on the current stream being a KeyedStream or not.

Parameters:

broadcastStream - The broadcast stream with the broadcast state to be connected with this stream.

Returns:

The BroadcastConnectedStream.

keyBy

public <K> KeyedStream<T,K> keyBy(KeySelector<T,K> key)

It creates a new KeyedStream that uses the provided key for partitioning its operator states.

Parameters:

key - The KeySelector to be used for extracting the key for partitioning

Returns:

The DataStream with partitioned state (i.e. KeyedStream)

keyBy

public <K> KeyedStream<T,K> keyBy(KeySelector<T,K> key,
                                  TypeInformation<K> keyType)

It creates a new KeyedStream that uses the provided key with explicit type information for partitioning its operator states.

Parameters:

key - The KeySelector to be used for extracting the key for partitioning.

keyType - The type information describing the key type.

Returns:

The DataStream with partitioned state (i.e. KeyedStream)

keyBy

@Deprecated
public KeyedStream<T,Tuple> keyBy(int... fields)

Deprecated. Use keyBy(KeySelector).

Partitions the operator state of a DataStream by the given key positions.

Parameters:

fields - The position of the fields on which the DataStream will be grouped.

Returns:

The DataStream with partitioned state (i.e. KeyedStream)

keyBy

@Deprecated
public KeyedStream<T,Tuple> keyBy(String... fields)

Deprecated. Use keyBy(KeySelector).

Partitions the operator state of a DataStream using field expressions. A field expression is either the name of a public field or a getter method with parentheses of the DataStream's underlying type. A dot can be used to drill down into objects, as in "field1.getInnerField2()" .

Parameters:

fields - One or more field expressions on which the state of the DataStream operators will be partitioned.

Returns:

The DataStream with partitioned state (i.e. KeyedStream)

partitionCustom

@Deprecated
public <K> DataStream<T> partitionCustom(Partitioner<K> partitioner,
                                                     int field)

Deprecated. use partitionCustom(Partitioner, KeySelector).

Partitions a tuple DataStream on the specified key fields using a custom partitioner. This method takes the key position to partition on, and a partitioner that accepts the key type.

Note: This method works only on single field keys.

Parameters:

partitioner - The partitioner to assign partitions to keys.

field - The field index on which the DataStream is partitioned.

Returns:

The partitioned DataStream.

partitionCustom

@Deprecated
public <K> DataStream<T> partitionCustom(Partitioner<K> partitioner,
                                                     String field)

Deprecated. use partitionCustom(Partitioner, KeySelector).

Partitions a POJO DataStream on the specified key fields using a custom partitioner. This method takes the key expression to partition on, and a partitioner that accepts the key type.

Note: This method works only on single field keys.

Parameters:

partitioner - The partitioner to assign partitions to keys.

field - The expression for the field on which the DataStream is partitioned.

Returns:

The partitioned DataStream.

partitionCustom

public <K> DataStream<T> partitionCustom(Partitioner<K> partitioner,
                                         KeySelector<T,K> keySelector)

Partitions a DataStream on the key returned by the selector, using a custom partitioner. This method takes the key selector to get the key to partition on, and a partitioner that accepts the key type.

Note: This method works only on single field keys, i.e. the selector cannot return tuples of fields.

Parameters:

partitioner - The partitioner to assign partitions to keys.

keySelector - The KeySelector with which the DataStream is partitioned.

Returns:

The partitioned DataStream.

See Also:

KeySelector

broadcast

public DataStream<T> broadcast()

Sets the partitioning of the DataStream so that the output elements are broadcasted to every parallel instance of the next operation.

Returns:

The DataStream with broadcast partitioning set.

broadcast

@PublicEvolving
public BroadcastStream<T> broadcast(MapStateDescriptor<?,?>... broadcastStateDescriptors)

Sets the partitioning of the DataStream so that the output elements are broadcasted to every parallel instance of the next operation. In addition, it implicitly as many broadcast states as the specified descriptors which can be used to store the element of the stream.

Parameters:

broadcastStateDescriptors - the descriptors of the broadcast states to create.

Returns:

A BroadcastStream which can be used in the connect(BroadcastStream) to create a BroadcastConnectedStream for further processing of the elements.

shuffle

@PublicEvolving
public DataStream<T> shuffle()

Sets the partitioning of the DataStream so that the output elements are shuffled uniformly randomly to the next operation.

Returns:

The DataStream with shuffle partitioning set.

forward

public DataStream<T> forward()

Sets the partitioning of the DataStream so that the output elements are forwarded to the local subtask of the next operation.

Returns:

The DataStream with forward partitioning set.

rebalance

public DataStream<T> rebalance()

Sets the partitioning of the DataStream so that the output elements are distributed evenly to instances of the next operation in a round-robin fashion.

Returns:

The DataStream with rebalance partitioning set.

rescale

@PublicEvolving
public DataStream<T> rescale()

Sets the partitioning of the DataStream so that the output elements are distributed evenly to a subset of instances of the next operation in a round-robin fashion.

The subset of downstream operations to which the upstream operation sends elements depends on the degree of parallelism of both the upstream and downstream operation. For example, if the upstream operation has parallelism 2 and the downstream operation has parallelism 4, then one upstream operation would distribute elements to two downstream operations while the other upstream operation would distribute to the other two downstream operations. If, on the other hand, the downstream operation has parallelism 2 while the upstream operation has parallelism 4 then two upstream operations will distribute to one downstream operation while the other two upstream operations will distribute to the other downstream operations.

In cases where the different parallelisms are not multiples of each other one or several downstream operations will have a differing number of inputs from upstream operations.

Returns:

The DataStream with rescale partitioning set.

global

@PublicEvolving
public DataStream<T> global()

Sets the partitioning of the DataStream so that the output values all go to the first instance of the next processing operator. Use this setting with care since it might cause a serious performance bottleneck in the application.

Returns:

The DataStream with shuffle partitioning set.

iterate

@Deprecated
public IterativeStream<T> iterate()

Deprecated. This method is deprecated since Flink 1.19. The only known use case of this Iteration API comes from Flink ML, which already has its own implementation of iteration and no longer uses this API. If there's any use cases other than Flink ML that needs iteration support, please reach out to dev@flink.apache.org and we can consider making the Flink ML iteration implementation a separate common library.

Initiates an iterative part of the program that feeds back data streams. The iterative part needs to be closed by calling IterativeStream.closeWith(DataStream). The transformation of this IterativeStream will be the iteration head. The data stream given to the IterativeStream.closeWith(DataStream) method is the data stream that will be fed back and used as the input for the iteration head. The user can also use different feedback type than the input of the iteration and treat the input and feedback streams as a ConnectedStreams be calling IterativeStream.withFeedbackType(TypeInformation)

A common usage pattern for streaming iterations is to use output splitting to send a part of the closing data stream to the head. Refer to ProcessFunction.Context#output(OutputTag, Object) for more information.

The iteration edge will be partitioned the same way as the first input of the iteration head unless it is changed in the IterativeStream.closeWith(DataStream) call.

By default a DataStream with iteration will never terminate, but the user can use the maxWaitTime parameter to set a max waiting time for the iteration head. If no data received in the set time, the stream terminates.

Returns:

The iterative data stream created.

See Also:

FLIP-357: Deprecate Iteration API of DataStream , Flink ML

iterate

@Deprecated
public IterativeStream<T> iterate(long maxWaitTimeMillis)

Deprecated. This method is deprecated since Flink 1.19. The only known use case of this Iteration API comes from Flink ML, which already has its own implementation of iteration and no longer uses this API. If there's any use cases other than Flink ML that needs iteration support, please reach out to dev@flink.apache.org and we can consider making the Flink ML iteration implementation a separate common library.

Initiates an iterative part of the program that feeds back data streams. The iterative part needs to be closed by calling IterativeStream.closeWith(DataStream). The transformation of this IterativeStream will be the iteration head. The data stream given to the IterativeStream.closeWith(DataStream) method is the data stream that will be fed back and used as the input for the iteration head. The user can also use different feedback type than the input of the iteration and treat the input and feedback streams as a ConnectedStreams be calling IterativeStream.withFeedbackType(TypeInformation)

A common usage pattern for streaming iterations is to use output splitting to send a part of the closing data stream to the head. Refer to ProcessFunction.Context#output(OutputTag, Object) for more information.

The iteration edge will be partitioned the same way as the first input of the iteration head unless it is changed in the IterativeStream.closeWith(DataStream) call.

By default a DataStream with iteration will never terminate, but the user can use the maxWaitTime parameter to set a max waiting time for the iteration head. If no data received in the set time, the stream terminates.

Parameters:

maxWaitTimeMillis - Number of milliseconds to wait between inputs before shutting down

Returns:

The iterative data stream created.

See Also:

FLIP-357: Deprecate Iteration API of DataStream , Flink ML

map

public <R> SingleOutputStreamOperator<R> map(MapFunction<T,R> mapper)

Applies a Map transformation on a DataStream. The transformation calls a MapFunction for each element of the DataStream. Each MapFunction call returns exactly one element. The user can also extend RichMapFunction to gain access to other features provided by the RichFunction interface.

Type Parameters:

R - output type

Parameters:

mapper - The MapFunction that is called for each element of the DataStream.

Returns:

The transformed DataStream.

map

public <R> SingleOutputStreamOperator<R> map(MapFunction<T,R> mapper,
                                             TypeInformation<R> outputType)

Applies a Map transformation on a DataStream. The transformation calls a MapFunction for each element of the DataStream. Each MapFunction call returns exactly one element. The user can also extend RichMapFunction to gain access to other features provided by the RichFunction interface.

Type Parameters:

R - output type

Parameters:

mapper - The MapFunction that is called for each element of the DataStream.

outputType - TypeInformation for the result type of the function.

Returns:

The transformed DataStream.

flatMap

public <R> SingleOutputStreamOperator<R> flatMap(FlatMapFunction<T,R> flatMapper)

Applies a FlatMap transformation on a DataStream. The transformation calls a FlatMapFunction for each element of the DataStream. Each FlatMapFunction call can return any number of elements including none. The user can also extend RichFlatMapFunction to gain access to other features provided by the RichFunction interface.

Type Parameters:

R - output type

Parameters:

flatMapper - The FlatMapFunction that is called for each element of the DataStream

Returns:

The transformed DataStream.

flatMap

public <R> SingleOutputStreamOperator<R> flatMap(FlatMapFunction<T,R> flatMapper,
                                                 TypeInformation<R> outputType)

Applies a FlatMap transformation on a DataStream. The transformation calls a FlatMapFunction for each element of the DataStream. Each FlatMapFunction call can return any number of elements including none. The user can also extend RichFlatMapFunction to gain access to other features provided by the RichFunction interface.

Type Parameters:

R - output type

Parameters:

flatMapper - The FlatMapFunction that is called for each element of the DataStream

outputType - TypeInformation for the result type of the function.

Returns:

The transformed DataStream.

process

@PublicEvolving
public <R> SingleOutputStreamOperator<R> process(ProcessFunction<T,R> processFunction)

Applies the given ProcessFunction on the input stream, thereby creating a transformed output stream.

The function will be called for every element in the input streams and can produce zero or more output elements.

Type Parameters:

R - The type of elements emitted by the ProcessFunction.

Parameters:

processFunction - The ProcessFunction that is called for each element in the stream.

Returns:

The transformed DataStream.

process

@Internal
public <R> SingleOutputStreamOperator<R> process(ProcessFunction<T,R> processFunction,
                                                           TypeInformation<R> outputType)

Applies the given ProcessFunction on the input stream, thereby creating a transformed output stream.

The function will be called for every element in the input streams and can produce zero or more output elements.

Type Parameters:

R - The type of elements emitted by the ProcessFunction.

Parameters:

processFunction - The ProcessFunction that is called for each element in the stream.

outputType - TypeInformation for the result type of the function.

Returns:

The transformed DataStream.

filter

public SingleOutputStreamOperator<T> filter(FilterFunction<T> filter)

Applies a Filter transformation on a DataStream. The transformation calls a FilterFunction for each element of the DataStream and retains only those element for which the function returns true. Elements for which the function returns false are filtered. The user can also extend RichFilterFunction to gain access to other features provided by the RichFunction interface.

Parameters:

filter - The FilterFunction that is called for each element of the DataStream.

Returns:

The filtered DataStream.

project

@PublicEvolving
public <R extends Tuple> SingleOutputStreamOperator<R> project(int... fieldIndexes)

Initiates a Project transformation on a Tuple DataStream.
Note: Only Tuple DataStreams can be projected.

The transformation projects each Tuple of the DataSet onto a (sub)set of fields.

Parameters:

fieldIndexes - The field indexes of the input tuples that are retained. The order of fields in the output tuple corresponds to the order of field indexes.

Returns:

The projected DataStream

See Also:

Tuple, DataStream

coGroup

public <T2> CoGroupedStreams<T,T2> coGroup(DataStream<T2> otherStream)

Creates a join operation. See CoGroupedStreams for an example of how the keys and window can be specified.

join

public <T2> JoinedStreams<T,T2> join(DataStream<T2> otherStream)

Creates a join operation. See JoinedStreams for an example of how the keys and window can be specified.

timeWindowAll

@Deprecated
public AllWindowedStream<T,TimeWindow> timeWindowAll(Time size)

Deprecated. Please use windowAll(WindowAssigner) with either TumblingEventTimeWindows or TumblingProcessingTimeWindows. For more information, see the deprecation notice on TimeCharacteristic

Windows this DataStream into tumbling time windows.

This is a shortcut for either .window(TumblingEventTimeWindows.of(size)) or .window(TumblingProcessingTimeWindows.of(size)) depending on the time characteristic set using

Note: This operation is inherently non-parallel since all elements have to pass through the same operator instance.

StreamExecutionEnvironment.setStreamTimeCharacteristic(org.apache.flink.streaming.api.TimeCharacteristic)

Parameters:

size - The size of the window.

timeWindowAll

@Deprecated
public AllWindowedStream<T,TimeWindow> timeWindowAll(Time size,
                                                                 Time slide)

Deprecated. Please use windowAll(WindowAssigner) with either SlidingEventTimeWindows or SlidingProcessingTimeWindows. For more information, see the deprecation notice on TimeCharacteristic

Windows this DataStream into sliding time windows.

This is a shortcut for either .window(SlidingEventTimeWindows.of(size, slide)) or .window(SlidingProcessingTimeWindows.of(size, slide)) depending on the time characteristic set using StreamExecutionEnvironment.setStreamTimeCharacteristic(org.apache.flink.streaming.api.TimeCharacteristic)

Note: This operation is inherently non-parallel since all elements have to pass through the same operator instance.

Parameters:

size - The size of the window.

countWindowAll

public AllWindowedStream<T,GlobalWindow> countWindowAll(long size)

Windows this DataStream into tumbling count windows.

Note: This operation is inherently non-parallel since all elements have to pass through the same operator instance.

Parameters:

size - The size of the windows in number of elements.

countWindowAll

public AllWindowedStream<T,GlobalWindow> countWindowAll(long size,
                                                        long slide)

Windows this DataStream into sliding count windows.

Note: This operation is inherently non-parallel since all elements have to pass through the same operator instance.

Parameters:

size - The size of the windows in number of elements.

slide - The slide interval in number of elements.

windowAll

@PublicEvolving
public <W extends Window> AllWindowedStream<T,W> windowAll(WindowAssigner<? super T,W> assigner)

Windows this data stream to a AllWindowedStream, which evaluates windows over a non key grouped stream. Elements are put into windows by a WindowAssigner. The grouping of elements is done by window.

A Trigger can be defined to specify when windows are evaluated. However, WindowAssigners have a default Trigger that is used if a Trigger is not specified.

Note: This operation is inherently non-parallel since all elements have to pass through the same operator instance.

Parameters:

assigner - The WindowAssigner that assigns elements to windows.

Returns:

The trigger windows data stream.

assignTimestampsAndWatermarks

public SingleOutputStreamOperator<T> assignTimestampsAndWatermarks(WatermarkStrategy<T> watermarkStrategy)

Assigns timestamps to the elements in the data stream and generates watermarks to signal event time progress. The given WatermarkStrategy is used to create a TimestampAssigner and WatermarkGenerator.

For each element in the data stream, the TimestampAssigner.extractTimestamp(Object, long) method is called to assign an event timestamp.

For each event in the data stream, the WatermarkGenerator.onEvent(Object, long, WatermarkOutput) will be called.

Periodically (defined by the ExecutionConfig.getAutoWatermarkInterval()), the WatermarkGenerator.onPeriodicEmit(WatermarkOutput) method will be called.

Common watermark generation patterns can be found as static methods in the WatermarkStrategy class.

Parameters:

watermarkStrategy - The strategy to generate watermarks based on event timestamps.

Returns:

The stream after the transformation, with assigned timestamps and watermarks.

assignTimestampsAndWatermarks

@Deprecated
public SingleOutputStreamOperator<T> assignTimestampsAndWatermarks(AssignerWithPeriodicWatermarks<T> timestampAndWatermarkAssigner)

Deprecated. Please use assignTimestampsAndWatermarks(WatermarkStrategy) instead.

Assigns timestamps to the elements in the data stream and periodically creates watermarks to signal event time progress.

This method uses the deprecated watermark generator interfaces. Please switch to assignTimestampsAndWatermarks(WatermarkStrategy) to use the new interfaces instead. The new interfaces support watermark idleness and no longer need to differentiate between "periodic" and "punctuated" watermarks.

assignTimestampsAndWatermarks

@Deprecated
public SingleOutputStreamOperator<T> assignTimestampsAndWatermarks(AssignerWithPunctuatedWatermarks<T> timestampAndWatermarkAssigner)

Deprecated. Please use assignTimestampsAndWatermarks(WatermarkStrategy) instead.

Assigns timestamps to the elements in the data stream and creates watermarks based on events, to signal event time progress.

This method uses the deprecated watermark generator interfaces. Please switch to assignTimestampsAndWatermarks(WatermarkStrategy) to use the new interfaces instead. The new interfaces support watermark idleness and no longer need to differentiate between "periodic" and "punctuated" watermarks.

print

@PublicEvolving
public DataStreamSink<T> print()

Writes a DataStream to the standard output stream (stdout).

For each element of the DataStream the result of Object.toString() is written.

NOTE: This will print to stdout on the machine where the code is executed, i.e. the Flink worker.

Returns:

The closed DataStream.

printToErr

@PublicEvolving
public DataStreamSink<T> printToErr()

Writes a DataStream to the standard error stream (stderr).

For each element of the DataStream the result of Object.toString() is written.

NOTE: This will print to stderr on the machine where the code is executed, i.e. the Flink worker.

Returns:

The closed DataStream.

print

@PublicEvolving
public DataStreamSink<T> print(String sinkIdentifier)

Writes a DataStream to the standard output stream (stdout).

For each element of the DataStream the result of Object.toString() is written.

NOTE: This will print to stdout on the machine where the code is executed, i.e. the Flink worker.

Parameters:

sinkIdentifier - The string to prefix the output with.

Returns:

The closed DataStream.

printToErr

@PublicEvolving
public DataStreamSink<T> printToErr(String sinkIdentifier)

Writes a DataStream to the standard error stream (stderr).

For each element of the DataStream the result of Object.toString() is written.

NOTE: This will print to stderr on the machine where the code is executed, i.e. the Flink worker.

Parameters:

sinkIdentifier - The string to prefix the output with.

Returns:

The closed DataStream.

writeAsText

@Deprecated
 @PublicEvolving
public DataStreamSink<T> writeAsText(String path)

Deprecated. Please use the StreamingFileSink explicitly using the addSink(SinkFunction) method.

Writes a DataStream to the file specified by path in text format.

For every element of the DataStream the result of Object.toString() is written.

Parameters:

path - The path pointing to the location the text file is written to.

Returns:

The closed DataStream.

writeAsText

@Deprecated
 @PublicEvolving
public DataStreamSink<T> writeAsText(String path,
                                                                  FileSystem.WriteMode writeMode)

Deprecated. Please use the StreamingFileSink explicitly using the addSink(SinkFunction) method.

Writes a DataStream to the file specified by path in text format.

For every element of the DataStream the result of Object.toString() is written.

Parameters:

path - The path pointing to the location the text file is written to

writeMode - Controls the behavior for existing files. Options are NO_OVERWRITE and OVERWRITE.

Returns:

The closed DataStream.

writeAsCsv

@Deprecated
 @PublicEvolving
public DataStreamSink<T> writeAsCsv(String path)

Deprecated. Please use the StreamingFileSink explicitly using the addSink(SinkFunction) method.

Writes a DataStream to the file specified by the path parameter.

For every field of an element of the DataStream the result of Object.toString() is written. This method can only be used on data streams of tuples.

Parameters:

path - the path pointing to the location the text file is written to

Returns:

the closed DataStream

writeAsCsv

@Deprecated
 @PublicEvolving
public DataStreamSink<T> writeAsCsv(String path,
                                                                 FileSystem.WriteMode writeMode)

Deprecated. Please use the StreamingFileSink explicitly using the addSink(SinkFunction) method.

Writes a DataStream to the file specified by the path parameter.

For every field of an element of the DataStream the result of Object.toString() is written. This method can only be used on data streams of tuples.

Parameters:

path - the path pointing to the location the text file is written to

writeMode - Controls the behavior for existing files. Options are NO_OVERWRITE and OVERWRITE.

Returns:

the closed DataStream

writeAsCsv

@Deprecated
 @PublicEvolving
public <X extends Tuple> DataStreamSink<T> writeAsCsv(String path,
                                                                                   FileSystem.WriteMode writeMode,
                                                                                   String rowDelimiter,
                                                                                   String fieldDelimiter)

Deprecated. Please use the StreamingFileSink explicitly using the addSink(SinkFunction) method.

Writes a DataStream to the file specified by the path parameter. The writing is performed periodically every millis milliseconds.

For every field of an element of the DataStream the result of Object.toString() is written. This method can only be used on data streams of tuples.

Parameters:

path - the path pointing to the location the text file is written to

writeMode - Controls the behavior for existing files. Options are NO_OVERWRITE and OVERWRITE.

rowDelimiter - the delimiter for two rows

fieldDelimiter - the delimiter for two fields

Returns:

the closed DataStream

writeToSocket

@PublicEvolving
public DataStreamSink<T> writeToSocket(String hostName,
                                                       int port,
                                                       SerializationSchema<T> schema)

Writes the DataStream to a socket as a byte array. The format of the output is specified by a SerializationSchema.

Parameters:

hostName - host of the socket

port - port of the socket

schema - schema for serialization

Returns:

the closed DataStream

writeUsingOutputFormat

@Deprecated
 @PublicEvolving
public DataStreamSink<T> writeUsingOutputFormat(OutputFormat<T> format)

Deprecated. Please use the StreamingFileSink explicitly using the addSink(SinkFunction) method.

Writes the dataStream into an output, described by an OutputFormat.

The output is not participating in Flink's checkpointing!

For writing to a file system periodically, the use of the StreamingFileSink is recommended.

Parameters:

format - The output format

Returns:

The closed DataStream

transform

@PublicEvolving
public <R> SingleOutputStreamOperator<R> transform(String operatorName,
                                                                   TypeInformation<R> outTypeInfo,
                                                                   OneInputStreamOperator<T,R> operator)

Method for passing user defined operators along with the type information that will transform the DataStream.

Type Parameters:

R - type of the return stream

Parameters:

operatorName - name of the operator, for logging purposes

outTypeInfo - the output type of the operator

operator - the object containing the transformation logic

Returns:

the data stream constructed

See Also:

transform(String, TypeInformation, OneInputStreamOperatorFactory)

transform

@PublicEvolving
public <R> SingleOutputStreamOperator<R> transform(String operatorName,
                                                                   TypeInformation<R> outTypeInfo,
                                                                   OneInputStreamOperatorFactory<T,R> operatorFactory)

Method for passing user defined operators created by the given factory along with the type information that will transform the DataStream.

This method uses the rather new operator factories and should only be used when custom factories are needed.

Type Parameters:

R - type of the return stream

Parameters:

operatorName - name of the operator, for logging purposes

outTypeInfo - the output type of the operator

operatorFactory - the factory for the operator.

Returns:

the data stream constructed.

doTransform

protected <R> SingleOutputStreamOperator<R> doTransform(String operatorName,
                                                        TypeInformation<R> outTypeInfo,
                                                        StreamOperatorFactory<R> operatorFactory)

setConnectionType

protected DataStream<T> setConnectionType(StreamPartitioner<T> partitioner)

Internal function for setting the partitioner for the DataStream.

Parameters:

partitioner - Partitioner to set.

Returns:

The modified DataStream.

addSink

public DataStreamSink<T> addSink(SinkFunction<T> sinkFunction)

Adds the given sink to this DataStream. Only streams with sinks added will be executed once the StreamExecutionEnvironment.execute() method is called.

Parameters:

sinkFunction - The object containing the sink's invoke function.

Returns:

The closed DataStream.

sinkTo

@PublicEvolving
public DataStreamSink<T> sinkTo(Sink<T,?,?,?> sink)

Adds the given Sink to this DataStream. Only streams with sinks added will be executed once the StreamExecutionEnvironment.execute() method is called.

Parameters:

sink - The user defined sink.

Returns:

The closed DataStream.

sinkTo

@PublicEvolving
public DataStreamSink<T> sinkTo(Sink<T,?,?,?> sink,
                                                CustomSinkOperatorUidHashes customSinkOperatorUidHashes)

Adds the given Sink to this DataStream. Only streams with sinks added will be executed once the StreamExecutionEnvironment.execute() method is called.

This method is intended to be used only to recover a snapshot where no uids have been set before taking the snapshot.

Parameters:

sink - The user defined sink.

Returns:

The closed DataStream.

sinkTo

@PublicEvolving
public DataStreamSink<T> sinkTo(Sink<T> sink)

Adds the given Sink to this DataStream. Only streams with sinks added will be executed once the StreamExecutionEnvironment.execute() method is called.

Parameters:

sink - The user defined sink.

Returns:

The closed DataStream.

sinkTo

@PublicEvolving
public DataStreamSink<T> sinkTo(Sink<T> sink,
                                                CustomSinkOperatorUidHashes customSinkOperatorUidHashes)

Adds the given Sink to this DataStream. Only streams with sinks added will be executed once the StreamExecutionEnvironment.execute() method is called.

This method is intended to be used only to recover a snapshot where no uids have been set before taking the snapshot.

Parameters:

customSinkOperatorUidHashes - operator hashes to support state binding

sink - The user defined sink.

Returns:

The closed DataStream.

executeAndCollect

public CloseableIterator<T> executeAndCollect()
                                       throws Exception

Triggers the distributed execution of the streaming dataflow and returns an iterator over the elements of the given DataStream.

The DataStream application is executed in the regular distributed manner on the target environment, and the events from the stream are polled back to this application process and thread through Flink's REST API.

IMPORTANT The returned iterator must be closed to free all cluster resources.

Throws:

Exception

executeAndCollect

public CloseableIterator<T> executeAndCollect(String jobExecutionName)
                                       throws Exception

Triggers the distributed execution of the streaming dataflow and returns an iterator over the elements of the given DataStream.

The DataStream application is executed in the regular distributed manner on the target environment, and the events from the stream are polled back to this application process and thread through Flink's REST API.

IMPORTANT The returned iterator must be closed to free all cluster resources.

Throws:

Exception

executeAndCollect

public List<T> executeAndCollect(int limit)
                          throws Exception

Triggers the distributed execution of the streaming dataflow and returns an iterator over the elements of the given DataStream.

The DataStream application is executed in the regular distributed manner on the target environment, and the events from the stream are polled back to this application process and thread through Flink's REST API.

Throws:

Exception

executeAndCollect

public List<T> executeAndCollect(String jobExecutionName,
                                 int limit)
                          throws Exception

Triggers the distributed execution of the streaming dataflow and returns an iterator over the elements of the given DataStream.

The DataStream application is executed in the regular distributed manner on the target environment, and the events from the stream are polled back to this application process and thread through Flink's REST API.

Throws:

Exception

collectAsync

@Experimental
public CloseableIterator<T> collectAsync()

Sets up the collection of the elements in this DataStream, and returns an iterator over the collected elements that can be used to retrieve elements once the job execution has started.

Caution: When multiple streams are being collected it is recommended to consume all streams in parallel to not back-pressure the job.

Caution: Closing the returned iterator cancels the job! It is recommended to close all iterators once you are no longer interested in any of the collected streams.

This method is functionally equivalent to collectAsync(Collector).

Returns:

iterator over the contained elements

collectAsync

@Experimental
public void collectAsync(DataStream.Collector<T> collector)

Sets up the collection of the elements in this DataStream, which can be retrieved later via the given DataStream.Collector.

Caution: When multiple streams are being collected it is recommended to consume all streams in parallel to not back-pressure the job.

Caution: Closing the iterator from the collector cancels the job! It is recommended to close all iterators once you are no longer interested in any of the collected streams.

This method is functionally equivalent to collectAsync().

This method is meant to support use-cases where the application of a sink is done via a Consumer<DataStream<T>>, where it wouldn't be possible (or inconvenient) to return an iterator.

Parameters:

collector - a collector that can be used to retrieve the elements

fullWindowPartition

@PublicEvolving
public PartitionWindowedStream<T> fullWindowPartition()

Collect records from each partition into a separate full window. The window emission will be triggered at the end of inputs. For this non-keyed data stream(each record has no key), a partition contains all records of a subtask.

Returns:

The full windowed data stream on partition.

getTransformation

@Internal
public Transformation<T> getTransformation()

Returns the Transformation that represents the operation that logically creates this DataStream.

Returns:

The Transformation