T- Type of the elements in the Stream
@Internal public class ForwardForConsecutiveHashPartitioner<T> extends ForwardPartitioner<T>
A --[hash]--> B --[hash]--> C | V A --[hash]--> B --[forward]--> C
However, sometimes the consecutive hash operators are not chained (e.g. multiple inputs), and this kind of forward partitioners will turn into forward job edges. These forward edges still have the consecutive hash assumption, so that they cannot be changed into rescale/rebalance edges, otherwise it can lead to incorrect results. This prevents the adaptive batch scheduler from determining parallelism for other forward edge downstream job vertices(see FLINK-25046).
To solve it, we introduce the
ForwardForConsecutiveHashPartitioner. When SQL planner
optimizes the case of multiple consecutive and the same hash shuffles, it should use this
partitioner, and then the runtime framework will change it to forward/hash after the operator
A --[hash]--> B --[hash]--> C | V A --[hash]--> B --[ForwardForConsecutiveHash]--> C
This partitioner will be converted to following partitioners after the operator chain creation:
1. Be converted to
ForwardPartitioner if this partitioner is intra-chain.
2. Be converted to
hashPartitioner if this
partitioner is inter-chain.
This partitioner should only be used for SQL Batch jobs and when using AdaptiveBatchScheduler.
|Constructor and Description|
Create a new ForwardForConsecutiveHashPartitioner.
|Modifier and Type||Method and Description|
Defines the behavior of this partitioner, when downstream rescaled during recovery of in-flight data.
Returns the logical channel index, to which the given record should be written.
equals, hashCode, isBroadcast, setup
public SubtaskStateMapper getDownstreamSubtaskStateMapper()
public int selectChannel(SerializationDelegate<StreamRecord<T>> record)
record- the record to determine the output channels for.
Copyright © 2014–2022 The Apache Software Foundation. All rights reserved.