org.apache.flink.streaming.api.datastream

Class ConnectedStreams<IN1,IN2>

java.lang.Object

org.apache.flink.streaming.api.datastream.ConnectedStreams<IN1,IN2>

Type Parameters:

IN1 - Type of the first input data steam.

IN2 - Type of the second input data stream.

Direct Known Subclasses:

IterativeStream.ConnectedIterativeStreams

@Public
public class ConnectedStreams<IN1,IN2>
extends Object

ConnectedStreams represent two connected streams of (possibly) different data types. Connected streams are useful for cases where operations on one stream directly affect the operations on the other stream, usually via shared state between the streams.

An example for the use of connected streams would be to apply rules that change over time onto another stream. One of the connected streams has the rules, the other stream the elements to apply the rules to. The operation on the connected stream maintains the current set of rules in the state. It may receive either a rule update and update the state or a data element and apply the rules in the state to the element.

The connected stream can be conceptually viewed as a union stream of an Either type, that holds either the first stream's type or the second stream's type.

Field Summary

Fields

Modifier and Type	Field and Description
protected StreamExecutionEnvironment	environment
protected DataStream<IN1>	inputStream1
protected DataStream<IN2>	inputStream2

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	ConnectedStreams(StreamExecutionEnvironment env, DataStream<IN1> input1, DataStream<IN2> input2)

Method Summary

Modifier and Type	Method and Description
<R> SingleOutputStreamOperator<R>	flatMap(CoFlatMapFunction<IN1,IN2,R> coFlatMapper) Applies a CoFlatMap transformation on a ConnectedStreams and maps the output to a common type.
<R> SingleOutputStreamOperator<R>	flatMap(CoFlatMapFunction<IN1,IN2,R> coFlatMapper, TypeInformation<R> outputType) Applies a CoFlatMap transformation on a ConnectedStreams and maps the output to a common type.
StreamExecutionEnvironment	getExecutionEnvironment()
DataStream<IN1>	getFirstInput() Returns the first DataStream.
DataStream<IN2>	getSecondInput() Returns the second DataStream.
TypeInformation<IN1>	getType1() Gets the type of the first input.
TypeInformation<IN2>	getType2() Gets the type of the second input.
ConnectedStreams<IN1,IN2>	keyBy(int[] keyPositions1, int[] keyPositions2) KeyBy operation for connected data stream.
ConnectedStreams<IN1,IN2>	keyBy(int keyPosition1, int keyPosition2) KeyBy operation for connected data stream.
<KEY> ConnectedStreams<IN1,IN2>	keyBy(KeySelector<IN1,KEY> keySelector1, KeySelector<IN2,KEY> keySelector2) KeyBy operation for connected data stream.
<KEY> ConnectedStreams<IN1,IN2>	keyBy(KeySelector<IN1,KEY> keySelector1, KeySelector<IN2,KEY> keySelector2, TypeInformation<KEY> keyType) KeyBy operation for connected data stream.
ConnectedStreams<IN1,IN2>	keyBy(String[] fields1, String[] fields2) KeyBy operation for connected data stream using key expressions. the elements of input1 and input2 according to fields1 and fields2.
ConnectedStreams<IN1,IN2>	keyBy(String field1, String field2) KeyBy operation for connected data stream using key expressions.
<R> SingleOutputStreamOperator<R>	map(CoMapFunction<IN1,IN2,R> coMapper) Applies a CoMap transformation on a ConnectedStreams and maps the output to a common type.
<R> SingleOutputStreamOperator<R>	map(CoMapFunction<IN1,IN2,R> coMapper, TypeInformation<R> outputType) Applies a CoMap transformation on a ConnectedStreams and maps the output to a common type.
<R> SingleOutputStreamOperator<R>	process(CoProcessFunction<IN1,IN2,R> coProcessFunction) Applies the given CoProcessFunction on the connected input streams, thereby creating a transformed output stream.
<R> SingleOutputStreamOperator<R>	process(CoProcessFunction<IN1,IN2,R> coProcessFunction, TypeInformation<R> outputType) Applies the given CoProcessFunction on the connected input streams, thereby creating a transformed output stream.
<K,R> SingleOutputStreamOperator<R>	process(KeyedCoProcessFunction<K,IN1,IN2,R> keyedCoProcessFunction) Applies the given KeyedCoProcessFunction on the connected input keyed streams, thereby creating a transformed output stream.
<K,R> SingleOutputStreamOperator<R>	process(KeyedCoProcessFunction<K,IN1,IN2,R> keyedCoProcessFunction, TypeInformation<R> outputType) Applies the given KeyedCoProcessFunction on the connected input streams, thereby creating a transformed output stream.
<R> SingleOutputStreamOperator<R>	transform(String functionName, TypeInformation<R> outTypeInfo, TwoInputStreamOperator<IN1,IN2,R> operator)
<R> SingleOutputStreamOperator<R>	transform(String functionName, TypeInformation<R> outTypeInfo, TwoInputStreamOperatorFactory<IN1,IN2,R> operatorFactory)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

environment

protected final StreamExecutionEnvironment environment

inputStream1

protected final DataStream<IN1> inputStream1

inputStream2

protected final DataStream<IN2> inputStream2

Constructor Detail

ConnectedStreams

protected ConnectedStreams(StreamExecutionEnvironment env,
                           DataStream<IN1> input1,
                           DataStream<IN2> input2)

Method Detail

getExecutionEnvironment

public StreamExecutionEnvironment getExecutionEnvironment()

getFirstInput

public DataStream<IN1> getFirstInput()

Returns the first DataStream.

Returns:

The first DataStream.

getSecondInput

public DataStream<IN2> getSecondInput()

Returns the second DataStream.

Returns:

The second DataStream.

getType1

public TypeInformation<IN1> getType1()

Gets the type of the first input.

Returns:

The type of the first input

getType2

public TypeInformation<IN2> getType2()

Gets the type of the second input.

Returns:

The type of the second input

keyBy

public ConnectedStreams<IN1,IN2> keyBy(int keyPosition1,
                                       int keyPosition2)

KeyBy operation for connected data stream. Assigns keys to the elements of input1 and input2 according to keyPosition1 and keyPosition2.

Parameters:

keyPosition1 - The field used to compute the hashcode of the elements in the first input stream.

keyPosition2 - The field used to compute the hashcode of the elements in the second input stream.

Returns:

The grouped ConnectedStreams

keyBy

public ConnectedStreams<IN1,IN2> keyBy(int[] keyPositions1,
                                       int[] keyPositions2)

KeyBy operation for connected data stream. Assigns keys to the elements of input1 and input2 according to keyPositions1 and keyPositions2.

Parameters:

keyPositions1 - The fields used to group the first input stream.

keyPositions2 - The fields used to group the second input stream.

Returns:

The grouped ConnectedStreams

keyBy

public ConnectedStreams<IN1,IN2> keyBy(String field1,
                                       String field2)

KeyBy operation for connected data stream using key expressions. Assigns keys to the elements of input1 and input2 according to field1 and field2. A field expression is either the name of a public field or a getter method with parentheses of the DataStreamS underlying type. A dot can be used to drill down into objects, as in "field1.getInnerField2()" .

Parameters:

field1 - The grouping expression for the first input

field2 - The grouping expression for the second input

Returns:

The grouped ConnectedStreams

keyBy

public ConnectedStreams<IN1,IN2> keyBy(String[] fields1,
                                       String[] fields2)

KeyBy operation for connected data stream using key expressions. the elements of input1 and input2 according to fields1 and fields2. A field expression is either the name of a public field or a getter method with parentheses of the DataStreamS underlying type. A dot can be used to drill down into objects, as in "field1.getInnerField2()" .

Parameters:

fields1 - The grouping expressions for the first input

fields2 - The grouping expressions for the second input

Returns:

The grouped ConnectedStreams

keyBy

public <KEY> ConnectedStreams<IN1,IN2> keyBy(KeySelector<IN1,KEY> keySelector1,
                                             KeySelector<IN2,KEY> keySelector2)

KeyBy operation for connected data stream. Assigns keys to the elements of input1 and input2 using keySelector1 and keySelector2.

Parameters:

keySelector1 - The KeySelector used for grouping the first input

keySelector2 - The KeySelector used for grouping the second input

Returns:

The partitioned ConnectedStreams

keyBy

public <KEY> ConnectedStreams<IN1,IN2> keyBy(KeySelector<IN1,KEY> keySelector1,
                                             KeySelector<IN2,KEY> keySelector2,
                                             TypeInformation<KEY> keyType)

KeyBy operation for connected data stream. Assigns keys to the elements of input1 and input2 using keySelector1 and keySelector2 with explicit type information for the common key type.

Parameters:

keySelector1 - The KeySelector used for grouping the first input

keySelector2 - The KeySelector used for grouping the second input

keyType - The type information of the common key type.

Returns:

The partitioned ConnectedStreams

map

public <R> SingleOutputStreamOperator<R> map(CoMapFunction<IN1,IN2,R> coMapper)

Applies a CoMap transformation on a ConnectedStreams and maps the output to a common type. The transformation calls a CoMapFunction.map1(IN1) for each element of the first input and CoMapFunction.map2(IN2) for each element of the second input. Each CoMapFunction call returns exactly one element.

Parameters:

coMapper - The CoMapFunction used to jointly transform the two input DataStreams

Returns:

The transformed DataStream

map

public <R> SingleOutputStreamOperator<R> map(CoMapFunction<IN1,IN2,R> coMapper,
                                             TypeInformation<R> outputType)

Applies a CoMap transformation on a ConnectedStreams and maps the output to a common type. The transformation calls a CoMapFunction.map1(IN1) for each element of the first input and CoMapFunction.map2(IN2) for each element of the second input. Each CoMapFunction call returns exactly one element.

Parameters:

coMapper - The CoMapFunction used to jointly transform the two input DataStreams

outputType - TypeInformation for the result type of the function.

Returns:

The transformed DataStream

flatMap

public <R> SingleOutputStreamOperator<R> flatMap(CoFlatMapFunction<IN1,IN2,R> coFlatMapper)

Applies a CoFlatMap transformation on a ConnectedStreams and maps the output to a common type. The transformation calls a CoFlatMapFunction.flatMap1(IN1, org.apache.flink.util.Collector<OUT>) for each element of the first input and CoFlatMapFunction.flatMap2(IN2, org.apache.flink.util.Collector<OUT>) for each element of the second input. Each CoFlatMapFunction call returns any number of elements including none.

Parameters:

coFlatMapper - The CoFlatMapFunction used to jointly transform the two input DataStreams

Returns:

The transformed DataStream

flatMap

public <R> SingleOutputStreamOperator<R> flatMap(CoFlatMapFunction<IN1,IN2,R> coFlatMapper,
                                                 TypeInformation<R> outputType)

Applies a CoFlatMap transformation on a ConnectedStreams and maps the output to a common type. The transformation calls a CoFlatMapFunction.flatMap1(IN1, org.apache.flink.util.Collector<OUT>) for each element of the first input and CoFlatMapFunction.flatMap2(IN2, org.apache.flink.util.Collector<OUT>) for each element of the second input. Each CoFlatMapFunction call returns any number of elements including none.

Parameters:

coFlatMapper - The CoFlatMapFunction used to jointly transform the two input DataStreams

outputType - TypeInformation for the result type of the function.

Returns:

The transformed DataStream

process

@PublicEvolving
public <R> SingleOutputStreamOperator<R> process(CoProcessFunction<IN1,IN2,R> coProcessFunction)

Applies the given CoProcessFunction on the connected input streams, thereby creating a transformed output stream.

The function will be called for every element in the input streams and can produce zero or more output elements. Contrary to the flatMap(CoFlatMapFunction) function, this function can also query the time and set timers. When reacting to the firing of set timers the function can directly emit elements and/or register yet more timers.

Type Parameters:

R - The type of elements emitted by the CoProcessFunction.

Parameters:

coProcessFunction - The CoProcessFunction that is called for each element in the stream.

Returns:

The transformed DataStream.

process

@Internal
public <R> SingleOutputStreamOperator<R> process(CoProcessFunction<IN1,IN2,R> coProcessFunction,
                                                           TypeInformation<R> outputType)

Applies the given CoProcessFunction on the connected input streams, thereby creating a transformed output stream.

The function will be called for every element in the input streams and can produce zero or more output elements. Contrary to the flatMap(CoFlatMapFunction) function, this function can also query the time and set timers. When reacting to the firing of set timers the function can directly emit elements and/or register yet more timers.

Type Parameters:

R - The type of elements emitted by the CoProcessFunction.

Parameters:

coProcessFunction - The CoProcessFunction that is called for each element in the stream.

Returns:

The transformed DataStream.

process

@PublicEvolving
public <K,R> SingleOutputStreamOperator<R> process(KeyedCoProcessFunction<K,IN1,IN2,R> keyedCoProcessFunction)

Applies the given KeyedCoProcessFunction on the connected input keyed streams, thereby creating a transformed output stream.

The function will be called for every element in the input keyed streams and can produce zero or more output elements. Contrary to the flatMap(CoFlatMapFunction) function, this function can also query the time and set timers. When reacting to the firing of set timers the function can directly emit elements and/or register yet more timers.

Type Parameters:

R - The type of elements emitted by the CoProcessFunction.

Parameters:

keyedCoProcessFunction - The KeyedCoProcessFunction that is called for each element in the stream.

Returns:

The transformed DataStream.

process

@Internal
public <K,R> SingleOutputStreamOperator<R> process(KeyedCoProcessFunction<K,IN1,IN2,R> keyedCoProcessFunction,
                                                             TypeInformation<R> outputType)

Applies the given KeyedCoProcessFunction on the connected input streams, thereby creating a transformed output stream.

The function will be called for every element in the input streams and can produce zero or more output elements. Contrary to the flatMap(CoFlatMapFunction) function, this function can also query the time and set timers. When reacting to the firing of set timers the function can directly emit elements and/or register yet more timers.

Type Parameters:

R - The type of elements emitted by the CoProcessFunction.

Parameters:

keyedCoProcessFunction - The KeyedCoProcessFunction that is called for each element in the stream.

Returns:

The transformed DataStream.

transform

@PublicEvolving
public <R> SingleOutputStreamOperator<R> transform(String functionName,
                                                                   TypeInformation<R> outTypeInfo,
                                                                   TwoInputStreamOperator<IN1,IN2,R> operator)

transform

@PublicEvolving
public <R> SingleOutputStreamOperator<R> transform(String functionName,
                                                                   TypeInformation<R> outTypeInfo,
                                                                   TwoInputStreamOperatorFactory<IN1,IN2,R> operatorFactory)