public class NettyShuffleMaster extends Object implements ShuffleMaster<NettyShuffleDescriptor>
ShuffleMaster
for netty and local file based shuffle implementation.Constructor and Description |
---|
NettyShuffleMaster(Configuration conf) |
Modifier and Type | Method and Description |
---|---|
MemorySize |
computeShuffleMemorySizeForTask(TaskInputsOutputsDescriptor desc)
JM announces network memory requirement from the calculating result of this method.
|
CompletableFuture<Collection<PartitionWithMetrics>> |
getAllPartitionWithMetrics(JobID jobId)
Get all partitions and their metrics, the metrics include sizes of sub-partitions in a result
partition.
|
void |
registerJob(JobShuffleContext context)
Registers the target job together with the corresponding
JobShuffleContext to this
shuffle master. |
CompletableFuture<NettyShuffleDescriptor> |
registerPartitionWithProducer(JobID jobID,
PartitionDescriptor partitionDescriptor,
ProducerDescriptor producerDescriptor)
Asynchronously register a partition and its producer with the shuffle service.
|
void |
releasePartitionExternally(ShuffleDescriptor shuffleDescriptor)
Release any external resources occupied by the given partition.
|
void |
unregisterJob(JobID jobId)
Unregisters the target job from this shuffle master, which means the corresponding job has
reached a global termination state and all the allocated resources except for the cluster
partitions can be cleared.
|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
close, start
public NettyShuffleMaster(Configuration conf)
public CompletableFuture<NettyShuffleDescriptor> registerPartitionWithProducer(JobID jobID, PartitionDescriptor partitionDescriptor, ProducerDescriptor producerDescriptor)
ShuffleMaster
The returned shuffle descriptor is an internal handle which identifies the partition internally within the shuffle service. The descriptor should provide enough information to read from or write data to the partition.
registerPartitionWithProducer
in interface ShuffleMaster<NettyShuffleDescriptor>
jobID
- job ID of the corresponding job which registered the partitionpartitionDescriptor
- general job graph information about the partitionproducerDescriptor
- general producer information (location, execution id, connection
info)public void releasePartitionExternally(ShuffleDescriptor shuffleDescriptor)
ShuffleMaster
This call triggers release of any resources which are occupied by the given partition in
the external systems outside of the producer executor. This is mostly relevant for the batch
jobs and blocking result partitions. The producer local resources are managed by ShuffleDescriptor.storesLocalResourcesOn()
and ShuffleEnvironment.releasePartitionsLocally(Collection)
.
releasePartitionExternally
in interface ShuffleMaster<NettyShuffleDescriptor>
shuffleDescriptor
- shuffle descriptor of the result partition to release externally.public MemorySize computeShuffleMemorySizeForTask(TaskInputsOutputsDescriptor desc)
NettyShuffleEnvironmentOptions.NETWORK_BUFFERS_PER_CHANNEL
and
NettyShuffleEnvironmentOptions.NETWORK_EXTRA_BUFFERS_PER_GATE
, which means we should
always keep the consistency of configurations between JM, RM and TM in fine-grained resource
management, thus to guarantee that the processes of memory announcing and allocating respect
each other.computeShuffleMemorySizeForTask
in interface ShuffleMaster<NettyShuffleDescriptor>
desc
- describes task inputs and outputs information for shuffle
memory calculation.TaskInputsOutputsDescriptor
.public CompletableFuture<Collection<PartitionWithMetrics>> getAllPartitionWithMetrics(JobID jobId)
ShuffleMaster
getAllPartitionWithMetrics
in interface ShuffleMaster<NettyShuffleDescriptor>
jobId
- ID of the target jobpublic void registerJob(JobShuffleContext context)
ShuffleMaster
JobShuffleContext
to this
shuffle master. Through the shuffle context, one can obtain some basic information like job
ID, job configuration. It enables ShuffleMaster to notify JobMaster about lost result
partitions, so that JobMaster can identify and reproduce unavailable partitions earlier.registerJob
in interface ShuffleMaster<NettyShuffleDescriptor>
context
- the corresponding shuffle context of the target job.public void unregisterJob(JobID jobId)
ShuffleMaster
unregisterJob
in interface ShuffleMaster<NettyShuffleDescriptor>
jobId
- ID of the target job to be unregistered.Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.