public class ForwardHashExchangeProcessor extends Object implements ExecNodeGraphProcessor
ExecNodeGraphProcessor
which finds all ExecNode
s that require hash
distribution, but its input has no hash Exchange node. Currently, the hash distribution is
satisfied by its none exchange inputs with FORWARD partitioner (the node and its input has the
same parallelism). Once the parallelism is changed, the FORWARD behavior will be broken, and the
result will be wrong.
In order to meet the needs of flexible parallelism changing by adaptive scheduler, a special
BatchExecExchange
(with KEEP_INPUT_AS_IS distribution flag) will be added for the ExecNode
as its input. And then the StreamingJobGraphGenerator will decide which partitioner can
be used when dynamic-graph is enabled: FORWARD partitioner if nodes are chainable, else HASH
partitioner.
Its works only for batch job when dynamic-graph is enabled.
Constructor and Description |
---|
ForwardHashExchangeProcessor() |
Modifier and Type | Method and Description |
---|---|
ExecNodeGraph |
process(ExecNodeGraph execGraph,
ProcessorContext context)
Given an
ExecNodeGraph , process it and return the result ExecNodeGraph . |
public ExecNodeGraph process(ExecNodeGraph execGraph, ProcessorContext context)
ExecNodeGraphProcessor
ExecNodeGraph
, process it and return the result ExecNodeGraph
.process
in interface ExecNodeGraphProcessor
Copyright © 2014–2023 The Apache Software Foundation. All rights reserved.