public interface ShuffleDescriptor extends Serializable
The descriptor is used for the deployment of the partition producer/consumer and their data exchange
|Modifier and Type||Method and Description|
Returns whether the partition is known and registered with the
Returns the location of the producing task executor if the partition occupies local resources there.
default boolean isUnknown()
When a partition consumer is being scheduled, it can happen that the producer of the
partition (consumer input channel) has not been scheduled and its location and other relevant
data is yet to be defined. To proceed with the consumer deployment, currently unknown input
channels have to be marked with placeholders. The placeholder is a special implementation of
the shuffle descriptor:
Note: this method is not supposed to be overridden in concrete shuffle implementation. The
only class where it returns
Indicates that this partition occupies local resources in the producing task executor.
Such partition requires that the task executor is running and being connected to be able to
consume the produced data. This is mostly relevant for the batch jobs and blocking result
partitions which can outlive the producer lifetime and be released externally.
ShuffleEnvironment.releasePartitionsLocally(Collection) can be used to release such kind of
Copyright © 2014–2023 The Apache Software Foundation. All rights reserved.