public class ForwardHashExchangeProcessor extends Object implements ExecNodeGraphProcessor
ExecNodeGraphProcessorwhich finds all
ExecNodes that require hash distribution, but its input has no hash Exchange node. Currently, the hash distribution is satisfied by its none exchange inputs with FORWARD partitioner (the node and its input has the same parallelism). Once the parallelism is changed, the FORWARD behavior will be broken, and the result will be wrong.
In order to meet the needs of flexible parallelism changing by adaptive scheduler, a special
BatchExecExchange (with KEEP_INPUT_AS_IS distribution flag) will be added for the
ExecNode as its input. And then the StreamingJobGraphGenerator will decide which partitioner can
be used when dynamic-graph is enabled: FORWARD partitioner if nodes are chainable, else HASH
Its works only for batch job when dynamic-graph is enabled.
|Constructor and Description|
|Modifier and Type||Method and Description|
Copyright © 2014–2023 The Apache Software Foundation. All rights reserved.