DataStream

DataStream

A DataStream represents a stream of elements of the same type.

DataStream.get_name()	Gets the name of the current data stream.
DataStream.name(name)	Sets the name of the current data stream.
DataStream.uid(uid)	Sets an ID for this operator.
DataStream.set_uid_hash(uid_hash)	Sets an user provided hash for this operator.
DataStream.set_parallelism(parallelism)	Sets the parallelism for this operator.
DataStream.set_max_parallelism(max_parallelism)	Sets the maximum parallelism of this operator.
DataStream.get_type()	Gets the type of the stream.
DataStream.get_execution_environment()	Returns the StreamExecutionEnvironment that was used to create this DataStream.
DataStream.force_non_parallel()	Sets the parallelism and maximum parallelism of this operator to one.
DataStream.set_buffer_timeout(timeout_millis)	Sets the buffering timeout for data produced by this operation.
DataStream.start_new_chain()	Starts a new task chain beginning at this operator.
DataStream.disable_chaining()	Turns off chaining for this operator so thread co-location will not be used as an optimization.
DataStream.slot_sharing_group(slot_sharing_group)	Sets the slot sharing group of this operation.
DataStream.set_description(description)	Sets the description for this operator.
DataStream.map(func[, output_type])	Applies a Map transformation on a DataStream.
DataStream.flat_map(func[, output_type])	Applies a FlatMap transformation on a DataStream.
DataStream.key_by(key_selector[, key_type])	Creates a new KeyedStream that uses the provided key for partitioning its operator states.
DataStream.filter(func)	Applies a Filter transformation on a DataStream.
DataStream.union(*streams)	Creates a new DataStream by merging DataStream outputs of the same type with each other.
DataStream.connect(ds)	Creates a new 'ConnectedStreams' by connecting 'DataStream' outputs of (possible) different types with each other.
DataStream.shuffle()	Sets the partitioning of the DataStream so that the output elements are shuffled uniformly randomly to the next operation.
DataStream.project(*field_indexes)	Initiates a Project transformation on a Tuple DataStream.
DataStream.rescale()	Sets the partitioning of the DataStream so that the output elements are distributed evenly to a subset of instances of the next operation in a round-robin fashion.
DataStream.rebalance()	Sets the partitioning of the DataStream so that the output elements are distributed evenly to instances of the next operation in a round-robin fashion.
DataStream.forward()	Sets the partitioning of the DataStream so that the output elements are forwarded to the local sub-task of the next operation.
DataStream.broadcast()	Sets the partitioning of the DataStream so that the output elements are broadcasted to every parallel instance of the next operation.
DataStream.process(func[, output_type])	Applies the given ProcessFunction on the input stream, thereby creating a transformed output stream.
DataStream.assign_timestamps_and_watermarks(...)	Assigns timestamps to the elements in the data stream and generates watermarks to signal event time progress.
DataStream.partition_custom(partitioner, ...)	Partitions a DataStream on the key returned by the selector, using a custom partitioner.
DataStream.add_sink(sink_func)	Adds the given sink to this DataStream.
DataStream.sink_to(sink)	Adds the given sink to this DataStream.
DataStream.execute_and_collect([...])	Triggers the distributed execution of the streaming dataflow and returns an iterator over the elements of the given DataStream.
DataStream.print([sink_identifier])	Writes a DataStream to the standard output stream (stdout).

DataStreamSink

A Stream Sink. This is used for emitting elements from a streaming topology.

DataStreamSink.name(name)	Sets the name of this sink.
DataStreamSink.uid(uid)	Sets an ID for this operator.
DataStreamSink.set_uid_hash(uid_hash)	Sets an user provided hash for this operator.
DataStreamSink.set_parallelism(parallelism)	Sets the parallelism for this operator.
DataStreamSink.set_description(description)	Sets the description for this sink.
DataStreamSink.disable_chaining()	Turns off chaining for this operator so thread co-location will not be used as an optimization.
DataStreamSink.slot_sharing_group(...)	Sets the slot sharing group of this operation.

KeyedStream

A Stream Sink. This is used for emitting elements from a streaming topology.

KeyedStream.map(func[, output_type])	Applies a Map transformation on a KeyedStream.
KeyedStream.flat_map(func[, output_type])	Applies a FlatMap transformation on a KeyedStream.
KeyedStream.reduce(func)	Applies a reduce transformation on the grouped data stream grouped on by the given key position.
KeyedStream.filter(func)	Applies a Filter transformation on a DataStream.
KeyedStream.add_sink(sink_func)	Adds the given sink to this DataStream.
KeyedStream.key_by(key_selector[, key_type])	Creates a new KeyedStream that uses the provided key for partitioning its operator states.
KeyedStream.process(func[, output_type])	Applies the given ProcessFunction on the input stream, thereby creating a transformed output stream.
KeyedStream.window(window_assigner)	Windows this data stream to a WindowedStream, which evaluates windows over a key grouped stream.
KeyedStream.union(*streams)	Creates a new DataStream by merging DataStream outputs of the same type with each other.
KeyedStream.connect(ds)	Creates a new 'ConnectedStreams' by connecting 'DataStream' outputs of (possible) different types with each other.
KeyedStream.partition_custom(partitioner, ...)	Partitions a DataStream on the key returned by the selector, using a custom partitioner.
KeyedStream.print([sink_identifier])	Writes a DataStream to the standard output stream (stdout).

WindowedStream

A WindowedStream represents a data stream where elements are grouped by key, and for each key, the stream of elements is split into windows based on a WindowAssigner. Window emission is triggered based on a Trigger.

The windows are conceptually evaluated for each key individually, meaning windows can trigger at different points for each key.

Note that the WindowedStream is purely an API construct, during runtime the WindowedStream will be collapsed together with the KeyedStream and the operation over the window into one single operation.

WindowedStream.get_execution_environment()
WindowedStream.get_input_type()
WindowedStream.trigger(trigger)	Sets the Trigger that should be used to trigger window emission.
WindowedStream.allowed_lateness(time_ms)	Sets the time by which elements are allowed to be late.
WindowedStream.apply(window_function[, ...])	Applies the given window function to each window.
WindowedStream.process(process_window_function)	Applies the given window function to each window.

ConnectedStreams

ConnectedStreams represent two connected streams of (possibly) different data types. Connected streams are useful for cases where operations on one stream directly affect the operations on the other stream, usually via shared state between the streams.

An example for the use of connected streams would be to apply rules that change over time onto another stream. One of the connected streams has the rules, the other stream the elements to apply the rules to. The operation on the connected stream maintains the current set of rules in the state. It may receive either a rule update and update the state or a data element and apply the rules in the state to the element.

The connected stream can be conceptually viewed as a union stream of an Either type, that holds either the first stream’s type or the second stream’s type.

ConnectedStreams.key_by(key_selector1, ...)	KeyBy operation for connected data stream.
ConnectedStreams.map(func[, output_type])	Applies a CoMap transformation on a ConnectedStreams and maps the output to a common type.
ConnectedStreams.flat_map(func[, output_type])	Applies a CoFlatMap transformation on a ConnectedStreams and maps the output to a common type.
ConnectedStreams.process(func[, output_type])