public class NettyShuffleMaster extends Object implements ShuffleMaster<NettyShuffleDescriptor>
ShuffleMasterfor netty and local file based shuffle implementation.
|Constructor and Description|
|Modifier and Type||Method and Description|
JM announces network memory requirement from the calculating result of this method.
Asynchronously register a partition and its producer with the shuffle service.
Release any external resources occupied by the given partition.
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
close, registerJob, start, unregisterJob
public NettyShuffleMaster(Configuration conf)
public CompletableFuture<NettyShuffleDescriptor> registerPartitionWithProducer(JobID jobID, PartitionDescriptor partitionDescriptor, ProducerDescriptor producerDescriptor)
The returned shuffle descriptor is an internal handle which identifies the partition internally within the shuffle service. The descriptor should provide enough information to read from or write data to the partition.
jobID- job ID of the corresponding job which registered the partition
partitionDescriptor- general job graph information about the partition
producerDescriptor- general producer information (location, execution id, connection info)
public void releasePartitionExternally(ShuffleDescriptor shuffleDescriptor)
This call triggers release of any resources which are occupied by the given partition in
the external systems outside of the producer executor. This is mostly relevant for the batch
jobs and blocking result partitions. The producer local resources are managed by
shuffleDescriptor- shuffle descriptor of the result partition to release externally.
public MemorySize computeShuffleMemorySizeForTask(TaskInputsOutputsDescriptor desc)
NettyShuffleEnvironmentOptions.NETWORK_EXTRA_BUFFERS_PER_GATE, which means we should always keep the consistency of configurations between JM, RM and TM in fine-grained resource management, thus to guarantee that the processes of memory announcing and allocating respect each other.
desc- describes task inputs and outputs information for shuffle memory calculation.
Copyright © 2014–2023 The Apache Software Foundation. All rights reserved.