P- type of provided result partition writers
G- type of provided input gates
public interface ShuffleEnvironment<P extends ResultPartitionWriter,G extends IndexedInputGate> extends AutoCloseable
Input/Output interface of local shuffle service environment is based on memory
Buffers. A producer can write shuffle data into the buffers, obtained from the created
ResultPartitionWriters and a consumer reads the buffers from the created
The interface contains method's to manage the lifecycle of the local shuffle service environment:
start()must be called before using the shuffle service environment.
AutoCloseable.close()is called to release the shuffle service environment.
The interface implements a factory of result partition writers to produce shuffle data:
createResultPartitionWriters(org.apache.flink.runtime.shuffle.ShuffleIOOwnerContext, java.util.List<org.apache.flink.runtime.deployment.ResultPartitionDeploymentDescriptor>). The created writers are grouped per owner. The
owner is responsible for the writers' lifecycle from the moment of creation.
Partitions are fully released in the following cases:
ResultPartitionWriter.close()are called if the production has failed.
ResultPartitionWriter.close()have been called). Only one consumption attempt is ever expected for the PIPELINED partition at the moment so it can be released afterwards.
ShuffleDescriptor.storesLocalResourcesOn()) then also
The partitions, which currently still occupy local resources, can be queried with
The interface implements a factory for the input gates:
createInputGates(org.apache.flink.runtime.shuffle.ShuffleIOOwnerContext, org.apache.flink.runtime.io.network.partition.PartitionProducerStateProvider, java.util.List<org.apache.flink.runtime.deployment.InputGateDeploymentDescriptor>). The created gates are grouped per owner. The owner is
responsible for the gates' lifecycle from the moment of creation.
When the input gates are created, it can happen that not all consumed partitions are known at
that moment e.g. because their producers have not been started yet. Therefore, the
ShuffleEnvironment provides a method
updatePartitionInfo(org.apache.flink.runtime.executiongraph.ExecutionAttemptID, org.apache.flink.runtime.executiongraph.PartitionInfo) to update
them externally, when the producer becomes known. The update mechanism has to be threadsafe
because the updated gate can be read concurrently from a different thread.
|Modifier and Type||Method and Description|
Factory method for the
Factory method for the
Create a context of the shuffle input/output owner used to create partitions or gates belonging to the owner.
Report partitions which still occupy some resources locally.
Release local resources occupied by the given partitions.
Start the internal related services before using the shuffle service environment.
Update a gate with the newly available partition information, previously unknown.
int start() throws IOException
ShuffleIOOwnerContext createShuffleIOOwnerContext(String ownerName, ExecutionAttemptID executionAttemptID, MetricGroup parentGroup)
This method has to be called only once to avoid duplicated internal metric group registration.
ownerName- the owner name, used for logs
executionAttemptID- execution attempt id of the producer or consumer
parentGroup- parent of shuffle specific metric group
List<P> createResultPartitionWriters(ShuffleIOOwnerContext ownerContext, List<ResultPartitionDeploymentDescriptor> resultPartitionDeploymentDescriptors)
ResultPartitionWritersto produce result partitions.
The order of the
ResultPartitionWriters in the returned
collection should be the same as the iteration order of the passed
ownerContext- the owner context relevant for partition creation
resultPartitionDeploymentDescriptors- descriptors of the partition, produced by the owner
void releasePartitionsLocally(Collection<ResultPartitionID> partitionIds)
This is called for partitions which occupy resources locally (can be checked by
partitionIds- identifying the partitions to be released
List<G> createInputGates(ShuffleIOOwnerContext ownerContext, PartitionProducerStateProvider partitionProducerStateProvider, List<InputGateDeploymentDescriptor> inputGateDeploymentDescriptors)
InputGatesto consume result partitions.
The order of the
InputGates in the returned collection should be the
same as the iteration order of the passed
ownerContext- the owner context relevant for gate creation
partitionProducerStateProvider- producer state provider to query whether the producer is ready for consumption
inputGateDeploymentDescriptors- descriptors of the input gates to consume
boolean updatePartitionInfo(ExecutionAttemptID consumerID, PartitionInfo partitionInfo) throws IOException, InterruptedException
consumerID- execution id to distinguish gates with the same id from the different consumer executions
partitionInfo- information needed to consume the updated partition, e.g. network location
trueif the partition has been updated or
falseif the partition is not available anymore.
IOException- IO problem by the update
InterruptedException- potentially blocking operation was interrupted
Copyright © 2014–2023 The Apache Software Foundation. All rights reserved.