@Experimental public interface KeyedPartitionStream<K,T> extends DataStream
Modifier and Type | Interface and Description |
---|---|
static interface |
KeyedPartitionStream.TwoKeyedPartitionStreams<K,T1,T2>
This class represents a combination of two
KeyedPartitionStream . |
Modifier and Type | Method and Description |
---|---|
BroadcastStream<T> |
broadcast()
Transform this stream to a new
BroadcastStream . |
<T_OTHER,OUT> |
connectAndProcess(BroadcastStream<T_OTHER> other,
TwoInputBroadcastStreamProcessFunction<T,T_OTHER,OUT> processFunction)
Apply a two input operation to this and other
BroadcastStream . |
<T_OTHER,OUT> |
connectAndProcess(BroadcastStream<T_OTHER> other,
TwoInputBroadcastStreamProcessFunction<T,T_OTHER,OUT> processFunction,
KeySelector<OUT,K> newKeySelector)
Apply a two input operation to this and other
BroadcastStream . |
<T_OTHER,OUT> |
connectAndProcess(KeyedPartitionStream<K,T_OTHER> other,
TwoInputNonBroadcastStreamProcessFunction<T,T_OTHER,OUT> processFunction)
Apply a two input operation to this and other
KeyedPartitionStream . |
<T_OTHER,OUT> |
connectAndProcess(KeyedPartitionStream<K,T_OTHER> other,
TwoInputNonBroadcastStreamProcessFunction<T,T_OTHER,OUT> processFunction,
KeySelector<OUT,K> newKeySelector)
Apply a two input operation to this and other
KeyedPartitionStream .The two keyed
streams must have the same partitions, otherwise it makes no sense to connect them. |
GlobalStream<T> |
global()
Coalesce this stream to a
GlobalStream . |
<NEW_KEY> KeyedPartitionStream<NEW_KEY,T> |
keyBy(KeySelector<T,NEW_KEY> keySelector)
Transform this stream to a new
KeyedPartitionStream . |
<OUT> NonKeyedPartitionStream<OUT> |
process(OneInputStreamProcessFunction<T,OUT> processFunction)
Apply an operation to this
KeyedPartitionStream . |
<OUT> KeyedPartitionStream<K,OUT> |
process(OneInputStreamProcessFunction<T,OUT> processFunction,
KeySelector<OUT,K> newKeySelector)
Apply an operation to this
KeyedPartitionStream . |
<OUT1,OUT2> |
process(TwoOutputStreamProcessFunction<T,OUT1,OUT2> processFunction)
Apply a two output operation to this
KeyedPartitionStream . |
<OUT1,OUT2> |
process(TwoOutputStreamProcessFunction<T,OUT1,OUT2> processFunction,
KeySelector<OUT1,K> keySelector1,
KeySelector<OUT2,K> keySelector2)
Apply a two output operation to this
KeyedPartitionStream . |
NonKeyedPartitionStream<T> |
shuffle()
Transform this stream to a new
NonKeyedPartitionStream , data will be shuffled between
these two streams. |
void |
toSink(Sink<T> sink) |
<OUT> KeyedPartitionStream<K,OUT> process(OneInputStreamProcessFunction<T,OUT> processFunction, KeySelector<OUT,K> newKeySelector)
KeyedPartitionStream
.
This method is used to avoid shuffle after applying the process function. It is required
that for the same record, the new KeySelector
must extract the same key as the
original KeySelector
on this KeyedPartitionStream
. Otherwise, the partition
of data will be messy.
processFunction
- to perform operation.newKeySelector
- to select the key after process.KeyedPartitionStream
with this operation.<OUT> NonKeyedPartitionStream<OUT> process(OneInputStreamProcessFunction<T,OUT> processFunction)
KeyedPartitionStream
.
Generally, apply an operation to a KeyedPartitionStream
will result in a NonKeyedPartitionStream
, and you can manually generate a KeyedPartitionStream
via
keyBy partitioning. In some cases, you can guarantee that the partition on which the data is
processed will not change, then you can use process(OneInputStreamProcessFunction,
KeySelector)
to avoid shuffling.
processFunction
- to perform operation.NonKeyedPartitionStream
with this operation.<OUT1,OUT2> KeyedPartitionStream.TwoKeyedPartitionStreams<K,OUT1,OUT2> process(TwoOutputStreamProcessFunction<T,OUT1,OUT2> processFunction, KeySelector<OUT1,K> keySelector1, KeySelector<OUT2,K> keySelector2)
KeyedPartitionStream
.
This method is used to avoid shuffle after applying the process function. It is required
that for the same record, these new two KeySelector
s must extract the same key as the
original KeySelector
s on this KeyedPartitionStream
. Otherwise, the partition
of data will be messy.
processFunction
- to perform two output operation.keySelector1
- to select the key of first output.keySelector2
- to select the key of second output.KeyedPartitionStream.TwoKeyedPartitionStreams
with this operation.<OUT1,OUT2> NonKeyedPartitionStream.TwoNonKeyedPartitionStreams<OUT1,OUT2> process(TwoOutputStreamProcessFunction<T,OUT1,OUT2> processFunction)
KeyedPartitionStream
.processFunction
- to perform two output operation.NonKeyedPartitionStream.TwoNonKeyedPartitionStreams
with this operation.<T_OTHER,OUT> NonKeyedPartitionStream<OUT> connectAndProcess(KeyedPartitionStream<K,T_OTHER> other, TwoInputNonBroadcastStreamProcessFunction<T,T_OTHER,OUT> processFunction)
KeyedPartitionStream
. The two keyed
streams must have the same partitions, otherwise it makes no sense to connect them.
Generally, concatenating two KeyedPartitionStream
will result in a NonKeyedPartitionStream
, and you can manually generate a KeyedPartitionStream
via
keyBy partitioning. In some cases, you can guarantee that the partition on which the data is
processed will not change, then you can use connectAndProcess(KeyedPartitionStream,
TwoInputNonBroadcastStreamProcessFunction, KeySelector)
to avoid shuffling.
other
- KeyedPartitionStream
to perform operation with two input.processFunction
- to perform operation.NonKeyedPartitionStream
with this operation.<T_OTHER,OUT> KeyedPartitionStream<K,OUT> connectAndProcess(KeyedPartitionStream<K,T_OTHER> other, TwoInputNonBroadcastStreamProcessFunction<T,T_OTHER,OUT> processFunction, KeySelector<OUT,K> newKeySelector)
KeyedPartitionStream
.The two keyed
streams must have the same partitions, otherwise it makes no sense to connect them.
This method is used to avoid shuffle after applying the process function. It is required
that for the same record, the new KeySelector
must extract the same key as the
original KeySelector
s on these two KeyedPartitionStream
s. Otherwise, the
partition of data will be messy.
other
- KeyedPartitionStream
to perform operation with two input.processFunction
- to perform operation.newKeySelector
- to select the key after process.KeyedPartitionStream
with this operation.<T_OTHER,OUT> NonKeyedPartitionStream<OUT> connectAndProcess(BroadcastStream<T_OTHER> other, TwoInputBroadcastStreamProcessFunction<T,T_OTHER,OUT> processFunction)
BroadcastStream
.
Generally, concatenating KeyedPartitionStream
and BroadcastStream
will
result in a NonKeyedPartitionStream
, and you can manually generate a KeyedPartitionStream
via keyBy partitioning. In some cases, you can guarantee that the
partition on which the data is processed will not change, then you can use connectAndProcess(BroadcastStream, TwoInputBroadcastStreamProcessFunction, KeySelector)
to
avoid shuffling.
processFunction
- to perform operation.<T_OTHER,OUT> KeyedPartitionStream<K,OUT> connectAndProcess(BroadcastStream<T_OTHER> other, TwoInputBroadcastStreamProcessFunction<T,T_OTHER,OUT> processFunction, KeySelector<OUT,K> newKeySelector)
BroadcastStream
.
This method is used to avoid shuffle after applying the process function. It is required
that for the record from non-broadcast input, the new KeySelector
must extract the
same key as the original KeySelector
s on the KeyedPartitionStream
. Otherwise,
the partition of data will be messy. As for the record from broadcast input, the output key
from keyed partition itself instead of the new key selector, so the data it outputs will not
affect the partition.
other
- BroadcastStream
to perform operation with two input.processFunction
- to perform operation.newKeySelector
- to select the key after process.KeyedPartitionStream
with this operation.GlobalStream<T> global()
GlobalStream
.<NEW_KEY> KeyedPartitionStream<NEW_KEY,T> keyBy(KeySelector<T,NEW_KEY> keySelector)
KeyedPartitionStream
.keySelector
- to decide how to map data to partition.NonKeyedPartitionStream<T> shuffle()
NonKeyedPartitionStream
, data will be shuffled between
these two streams.BroadcastStream<T> broadcast()
BroadcastStream
.BroadcastStream
.Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.