T
- Type of the elements in the Stream@Internal public class ForwardForConsecutiveHashPartitioner<T> extends ForwardPartitioner<T>
A --[hash]--> B --[hash]--> C
|
V
A --[hash]--> B --[forward]--> C
However, sometimes the consecutive hash operators are not chained (e.g. multiple inputs), and this kind of forward partitioners will turn into forward job edges. These forward edges still have the consecutive hash assumption, so that they cannot be changed into rescale/rebalance edges, otherwise it can lead to incorrect results. This prevents the adaptive batch scheduler from determining parallelism for other forward edge downstream job vertices(see FLINK-25046).
To solve it, we introduce the ForwardForConsecutiveHashPartitioner
. When SQL planner
optimizes the case of multiple consecutive and the same hash shuffles, it should use this
partitioner, and then the runtime framework will change it to forward/hash after the operator
chain creation.
A --[hash]--> B --[hash]--> C
|
V
A --[hash]--> B --[ForwardForConsecutiveHash]--> C
This partitioner will be converted to following partitioners after the operator chain creation:
1. Be converted to ForwardPartitioner
if this partitioner is intra-chain.
2. Be converted to hashPartitioner
if this
partitioner is inter-chain.
This partitioner should only be used for SQL Batch jobs and when using AdaptiveBatchScheduler.
numberOfChannels
Constructor and Description |
---|
ForwardForConsecutiveHashPartitioner(StreamPartitioner<T> hashPartitioner)
Create a new ForwardForConsecutiveHashPartitioner.
|
Modifier and Type | Method and Description |
---|---|
StreamPartitioner<T> |
copy() |
SubtaskStateMapper |
getDownstreamSubtaskStateMapper()
Defines the behavior of this partitioner, when downstream rescaled during recovery of
in-flight data.
|
StreamPartitioner<T> |
getHashPartitioner() |
boolean |
isPointwise() |
int |
selectChannel(SerializationDelegate<StreamRecord<T>> record)
Returns the logical channel index, to which the given record should be written.
|
getUpstreamSubtaskStateMapper, toString
equals, hashCode, isBroadcast, setup
public ForwardForConsecutiveHashPartitioner(StreamPartitioner<T> hashPartitioner)
hashPartitioner
- the HashPartitionerpublic StreamPartitioner<T> copy()
copy
in class ForwardPartitioner<T>
public SubtaskStateMapper getDownstreamSubtaskStateMapper()
StreamPartitioner
getDownstreamSubtaskStateMapper
in class ForwardPartitioner<T>
public boolean isPointwise()
isPointwise
in class ForwardPartitioner<T>
public int selectChannel(SerializationDelegate<StreamRecord<T>> record)
ChannelSelector
UnsupportedOperationException
).selectChannel
in interface ChannelSelector<SerializationDelegate<StreamRecord<T>>>
selectChannel
in class ForwardPartitioner<T>
record
- the record to determine the output channels for.public StreamPartitioner<T> getHashPartitioner()
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.