Class ConnectedStreams<IN1,IN2>
- java.lang.Object
-
- org.apache.flink.streaming.api.datastream.ConnectedStreams<IN1,IN2>
-
- Type Parameters:
IN1
- Type of the first input data steam.IN2
- Type of the second input data stream.
@Public public class ConnectedStreams<IN1,IN2> extends Object
ConnectedStreams represent two connected streams of (possibly) different data types. Connected streams are useful for cases where operations on one stream directly affect the operations on the other stream, usually via shared state between the streams.An example for the use of connected streams would be to apply rules that change over time onto another stream. One of the connected streams has the rules, the other stream the elements to apply the rules to. The operation on the connected stream maintains the current set of rules in the state. It may receive either a rule update and update the state or a data element and apply the rules in the state to the element.
The connected stream can be conceptually viewed as a union stream of an Either type, that holds either the first stream's type or the second stream's type.
-
-
Field Summary
Fields Modifier and Type Field Description protected StreamExecutionEnvironment
environment
protected DataStream<IN1>
inputStream1
protected DataStream<IN2>
inputStream2
-
Constructor Summary
Constructors Modifier Constructor Description protected
ConnectedStreams(StreamExecutionEnvironment env, DataStream<IN1> input1, DataStream<IN2> input2)
-
Method Summary
All Methods Instance Methods Concrete Methods Deprecated Methods Modifier and Type Method Description <R> SingleOutputStreamOperator<R>
flatMap(CoFlatMapFunction<IN1,IN2,R> coFlatMapper)
Applies a CoFlatMap transformation on aConnectedStreams
and maps the output to a common type.<R> SingleOutputStreamOperator<R>
flatMap(CoFlatMapFunction<IN1,IN2,R> coFlatMapper, TypeInformation<R> outputType)
Applies a CoFlatMap transformation on aConnectedStreams
and maps the output to a common type.StreamExecutionEnvironment
getExecutionEnvironment()
DataStream<IN1>
getFirstInput()
Returns the firstDataStream
.DataStream<IN2>
getSecondInput()
Returns the secondDataStream
.TypeInformation<IN1>
getType1()
Gets the type of the first input.TypeInformation<IN2>
getType2()
Gets the type of the second input.ConnectedStreams<IN1,IN2>
keyBy(int[] keyPositions1, int[] keyPositions2)
Deprecated.ConnectedStreams<IN1,IN2>
keyBy(int keyPosition1, int keyPosition2)
Deprecated.ConnectedStreams<IN1,IN2>
keyBy(String[] fields1, String[] fields2)
Deprecated.ConnectedStreams<IN1,IN2>
keyBy(String field1, String field2)
Deprecated.<KEY> ConnectedStreams<IN1,IN2>
keyBy(KeySelector<IN1,KEY> keySelector1, KeySelector<IN2,KEY> keySelector2)
KeyBy operation for connected data stream.<KEY> ConnectedStreams<IN1,IN2>
keyBy(KeySelector<IN1,KEY> keySelector1, KeySelector<IN2,KEY> keySelector2, TypeInformation<KEY> keyType)
KeyBy operation for connected data stream.<R> SingleOutputStreamOperator<R>
map(CoMapFunction<IN1,IN2,R> coMapper)
Applies a CoMap transformation on aConnectedStreams
and maps the output to a common type.<R> SingleOutputStreamOperator<R>
map(CoMapFunction<IN1,IN2,R> coMapper, TypeInformation<R> outputType)
Applies a CoMap transformation on aConnectedStreams
and maps the output to a common type.<R> SingleOutputStreamOperator<R>
process(CoProcessFunction<IN1,IN2,R> coProcessFunction)
Applies the givenCoProcessFunction
on the connected input streams, thereby creating a transformed output stream.<R> SingleOutputStreamOperator<R>
process(CoProcessFunction<IN1,IN2,R> coProcessFunction, TypeInformation<R> outputType)
Applies the givenCoProcessFunction
on the connected input streams, thereby creating a transformed output stream.<K,R>
SingleOutputStreamOperator<R>process(KeyedCoProcessFunction<K,IN1,IN2,R> keyedCoProcessFunction)
Applies the givenKeyedCoProcessFunction
on the connected input keyed streams, thereby creating a transformed output stream.<K,R>
SingleOutputStreamOperator<R>process(KeyedCoProcessFunction<K,IN1,IN2,R> keyedCoProcessFunction, TypeInformation<R> outputType)
Applies the givenKeyedCoProcessFunction
on the connected input streams, thereby creating a transformed output stream.<R> SingleOutputStreamOperator<R>
transform(String functionName, TypeInformation<R> outTypeInfo, TwoInputStreamOperator<IN1,IN2,R> operator)
<R> SingleOutputStreamOperator<R>
transform(String functionName, TypeInformation<R> outTypeInfo, TwoInputStreamOperatorFactory<IN1,IN2,R> operatorFactory)
-
-
-
Field Detail
-
environment
protected final StreamExecutionEnvironment environment
-
inputStream1
protected final DataStream<IN1> inputStream1
-
inputStream2
protected final DataStream<IN2> inputStream2
-
-
Constructor Detail
-
ConnectedStreams
protected ConnectedStreams(StreamExecutionEnvironment env, DataStream<IN1> input1, DataStream<IN2> input2)
-
-
Method Detail
-
getExecutionEnvironment
public StreamExecutionEnvironment getExecutionEnvironment()
-
getFirstInput
public DataStream<IN1> getFirstInput()
Returns the firstDataStream
.- Returns:
- The first DataStream.
-
getSecondInput
public DataStream<IN2> getSecondInput()
Returns the secondDataStream
.- Returns:
- The second DataStream.
-
getType1
public TypeInformation<IN1> getType1()
Gets the type of the first input.- Returns:
- The type of the first input
-
getType2
public TypeInformation<IN2> getType2()
Gets the type of the second input.- Returns:
- The type of the second input
-
keyBy
@Deprecated public ConnectedStreams<IN1,IN2> keyBy(int keyPosition1, int keyPosition2)
Deprecated.KeyBy operation for connected data stream. Assigns keys to the elements of input1 and input2 according to keyPosition1 and keyPosition2.- Parameters:
keyPosition1
- The field used to compute the hashcode of the elements in the first input stream.keyPosition2
- The field used to compute the hashcode of the elements in the second input stream.- Returns:
- The grouped
ConnectedStreams
-
keyBy
@Deprecated public ConnectedStreams<IN1,IN2> keyBy(int[] keyPositions1, int[] keyPositions2)
Deprecated.KeyBy operation for connected data stream. Assigns keys to the elements of input1 and input2 according to keyPositions1 and keyPositions2.- Parameters:
keyPositions1
- The fields used to group the first input stream.keyPositions2
- The fields used to group the second input stream.- Returns:
- The grouped
ConnectedStreams
-
keyBy
@Deprecated public ConnectedStreams<IN1,IN2> keyBy(String field1, String field2)
Deprecated.KeyBy operation for connected data stream using key expressions. Assigns keys to the elements of input1 and input2 according to field1 and field2. A field expression is either the name of a public field or a getter method with parentheses of theDataStream
S underlying type. A dot can be used to drill down into objects, as in"field1.getInnerField2()"
.- Parameters:
field1
- The grouping expression for the first inputfield2
- The grouping expression for the second input- Returns:
- The grouped
ConnectedStreams
-
keyBy
@Deprecated public ConnectedStreams<IN1,IN2> keyBy(String[] fields1, String[] fields2)
Deprecated.KeyBy operation for connected data stream using key expressions. the elements of input1 and input2 according to fields1 and fields2. A field expression is either the name of a public field or a getter method with parentheses of theDataStream
S underlying type. A dot can be used to drill down into objects, as in"field1.getInnerField2()"
.- Parameters:
fields1
- The grouping expressions for the first inputfields2
- The grouping expressions for the second input- Returns:
- The grouped
ConnectedStreams
-
keyBy
public <KEY> ConnectedStreams<IN1,IN2> keyBy(KeySelector<IN1,KEY> keySelector1, KeySelector<IN2,KEY> keySelector2)
KeyBy operation for connected data stream. Assigns keys to the elements of input1 and input2 using keySelector1 and keySelector2.- Parameters:
keySelector1
- TheKeySelector
used for grouping the first inputkeySelector2
- TheKeySelector
used for grouping the second input- Returns:
- The partitioned
ConnectedStreams
-
keyBy
public <KEY> ConnectedStreams<IN1,IN2> keyBy(KeySelector<IN1,KEY> keySelector1, KeySelector<IN2,KEY> keySelector2, TypeInformation<KEY> keyType)
KeyBy operation for connected data stream. Assigns keys to the elements of input1 and input2 using keySelector1 and keySelector2 with explicit type information for the common key type.- Parameters:
keySelector1
- TheKeySelector
used for grouping the first inputkeySelector2
- TheKeySelector
used for grouping the second inputkeyType
- The type information of the common key type.- Returns:
- The partitioned
ConnectedStreams
-
map
public <R> SingleOutputStreamOperator<R> map(CoMapFunction<IN1,IN2,R> coMapper)
Applies a CoMap transformation on aConnectedStreams
and maps the output to a common type. The transformation calls aCoMapFunction.map1(IN1)
for each element of the first input andCoMapFunction.map2(IN2)
for each element of the second input. Each CoMapFunction call returns exactly one element.- Parameters:
coMapper
- The CoMapFunction used to jointly transform the two input DataStreams- Returns:
- The transformed
DataStream
-
map
public <R> SingleOutputStreamOperator<R> map(CoMapFunction<IN1,IN2,R> coMapper, TypeInformation<R> outputType)
Applies a CoMap transformation on aConnectedStreams
and maps the output to a common type. The transformation calls aCoMapFunction.map1(IN1)
for each element of the first input andCoMapFunction.map2(IN2)
for each element of the second input. Each CoMapFunction call returns exactly one element.- Parameters:
coMapper
- The CoMapFunction used to jointly transform the two input DataStreamsoutputType
-TypeInformation
for the result type of the function.- Returns:
- The transformed
DataStream
-
flatMap
public <R> SingleOutputStreamOperator<R> flatMap(CoFlatMapFunction<IN1,IN2,R> coFlatMapper)
Applies a CoFlatMap transformation on aConnectedStreams
and maps the output to a common type. The transformation calls aCoFlatMapFunction.flatMap1(IN1, org.apache.flink.util.Collector<OUT>)
for each element of the first input andCoFlatMapFunction.flatMap2(IN2, org.apache.flink.util.Collector<OUT>)
for each element of the second input. Each CoFlatMapFunction call returns any number of elements including none.- Parameters:
coFlatMapper
- The CoFlatMapFunction used to jointly transform the two input DataStreams- Returns:
- The transformed
DataStream
-
flatMap
public <R> SingleOutputStreamOperator<R> flatMap(CoFlatMapFunction<IN1,IN2,R> coFlatMapper, TypeInformation<R> outputType)
Applies a CoFlatMap transformation on aConnectedStreams
and maps the output to a common type. The transformation calls aCoFlatMapFunction.flatMap1(IN1, org.apache.flink.util.Collector<OUT>)
for each element of the first input andCoFlatMapFunction.flatMap2(IN2, org.apache.flink.util.Collector<OUT>)
for each element of the second input. Each CoFlatMapFunction call returns any number of elements including none.- Parameters:
coFlatMapper
- The CoFlatMapFunction used to jointly transform the two input DataStreamsoutputType
-TypeInformation
for the result type of the function.- Returns:
- The transformed
DataStream
-
process
@PublicEvolving public <R> SingleOutputStreamOperator<R> process(CoProcessFunction<IN1,IN2,R> coProcessFunction)
Applies the givenCoProcessFunction
on the connected input streams, thereby creating a transformed output stream.The function will be called for every element in the input streams and can produce zero or more output elements. Contrary to the
flatMap(CoFlatMapFunction)
function, this function can also query the time and set timers. When reacting to the firing of set timers the function can directly emit elements and/or register yet more timers.- Type Parameters:
R
- The type of elements emitted by theCoProcessFunction
.- Parameters:
coProcessFunction
- TheCoProcessFunction
that is called for each element in the stream.- Returns:
- The transformed
DataStream
.
-
process
@Internal public <R> SingleOutputStreamOperator<R> process(CoProcessFunction<IN1,IN2,R> coProcessFunction, TypeInformation<R> outputType)
Applies the givenCoProcessFunction
on the connected input streams, thereby creating a transformed output stream.The function will be called for every element in the input streams and can produce zero or more output elements. Contrary to the
flatMap(CoFlatMapFunction)
function, this function can also query the time and set timers. When reacting to the firing of set timers the function can directly emit elements and/or register yet more timers.- Type Parameters:
R
- The type of elements emitted by theCoProcessFunction
.- Parameters:
coProcessFunction
- TheCoProcessFunction
that is called for each element in the stream.- Returns:
- The transformed
DataStream
.
-
process
@PublicEvolving public <K,R> SingleOutputStreamOperator<R> process(KeyedCoProcessFunction<K,IN1,IN2,R> keyedCoProcessFunction)
Applies the givenKeyedCoProcessFunction
on the connected input keyed streams, thereby creating a transformed output stream.The function will be called for every element in the input keyed streams and can produce zero or more output elements. Contrary to the
flatMap(CoFlatMapFunction)
function, this function can also query the time and set timers. When reacting to the firing of set timers the function can directly emit elements and/or register yet more timers.- Type Parameters:
R
- The type of elements emitted by theCoProcessFunction
.- Parameters:
keyedCoProcessFunction
- TheKeyedCoProcessFunction
that is called for each element in the stream.- Returns:
- The transformed
DataStream
.
-
process
@Internal public <K,R> SingleOutputStreamOperator<R> process(KeyedCoProcessFunction<K,IN1,IN2,R> keyedCoProcessFunction, TypeInformation<R> outputType)
Applies the givenKeyedCoProcessFunction
on the connected input streams, thereby creating a transformed output stream.The function will be called for every element in the input streams and can produce zero or more output elements. Contrary to the
flatMap(CoFlatMapFunction)
function, this function can also query the time and set timers. When reacting to the firing of set timers the function can directly emit elements and/or register yet more timers.- Type Parameters:
R
- The type of elements emitted by theCoProcessFunction
.- Parameters:
keyedCoProcessFunction
- TheKeyedCoProcessFunction
that is called for each element in the stream.- Returns:
- The transformed
DataStream
.
-
transform
@PublicEvolving public <R> SingleOutputStreamOperator<R> transform(String functionName, TypeInformation<R> outTypeInfo, TwoInputStreamOperator<IN1,IN2,R> operator)
-
transform
@PublicEvolving public <R> SingleOutputStreamOperator<R> transform(String functionName, TypeInformation<R> outTypeInfo, TwoInputStreamOperatorFactory<IN1,IN2,R> operatorFactory)
-
-