public class SingleInputGate extends IndexedInputGate
Each intermediate result is partitioned over its producing parallel subtasks; each of these partitions is furthermore partitioned into one or more subpartitions.
As an example, consider a map-reduce program, where the map operator produces data and the reduce operator consumes the produced data.
+-----+ +---------------------+ +--------+
| Map | = produce => | Intermediate Result | <= consume = | Reduce |
+-----+ +---------------------+ +--------+
When deploying such a program in parallel, the intermediate result will be partitioned over its producing parallel subtasks; each of these partitions is furthermore partitioned into one or more subpartitions.
Intermediate result
+-----------------------------------------+
| +----------------+ | +-----------------------+
+-------+ | +-------------+ +=> | Subpartition 1 | | <=======+=== | Input Gate | Reduce 1 |
| Map 1 | ==> | | Partition 1 | =| +----------------+ | | +-----------------------+
+-------+ | +-------------+ +=> | Subpartition 2 | | <==+ |
| +----------------+ | | | Subpartition request
| | | |
| +----------------+ | | |
+-------+ | +-------------+ +=> | Subpartition 1 | | <==+====+
| Map 2 | ==> | | Partition 2 | =| +----------------+ | | +-----------------------+
+-------+ | +-------------+ +=> | Subpartition 2 | | <==+======== | Input Gate | Reduce 2 |
| +----------------+ | +-----------------------+
+-----------------------------------------+
In the above example, two map subtasks produce the intermediate result in parallel, resulting in two partitions (Partition 1 and 2). Each of these partitions is further partitioned into two subpartitions -- one for each parallel reduce subtask.
InputGate.InputWithData<INPUT,DATA>
AvailabilityProvider.AvailabilityHelper
availabilityHelper
AVAILABLE
Constructor and Description |
---|
SingleInputGate(String owningTaskName,
int gateIndex,
IntermediateDataSetID consumedResultId,
ResultPartitionType consumedPartitionType,
int consumedSubpartitionIndex,
int numberOfInputChannels,
PartitionProducerStateProvider partitionProducerStateProvider,
SupplierWithException<BufferPool,IOException> bufferPoolFactory,
BufferDecompressor bufferDecompressor,
MemorySegmentProvider memorySegmentProvider) |
Modifier and Type | Method and Description |
---|---|
void |
assignExclusiveSegments()
Assign the exclusive buffers to all remote input channels directly for credit-based mode.
|
void |
close() |
BufferPool |
getBufferPool() |
InputChannel |
getChannel(int channelIndex)
Returns the channel of this gate.
|
CompletableFuture<Void> |
getCloseFuture() |
ResultPartitionType |
getConsumedPartitionType()
Returns the type of this input channel's consumed result partition.
|
int |
getGateIndex()
Returns the index of this input gate.
|
Map<IntermediateResultPartitionID,InputChannel> |
getInputChannels() |
Optional<BufferOrEvent> |
getNext()
Blocking call waiting for next
BufferOrEvent . |
int |
getNumberOfInputChannels() |
int |
getNumberOfQueuedBuffers() |
String |
getOwningTaskName() |
boolean |
isFinished() |
Optional<BufferOrEvent> |
pollNext()
Poll the
BufferOrEvent . |
CompletableFuture<?> |
readRecoveredState(ExecutorService executor,
ChannelStateReader reader)
Reads the previous unaligned checkpoint states before requesting partition data.
|
void |
registerBufferReceivedListener(BufferReceivedListener bufferReceivedListener) |
void |
requestPartitions() |
void |
resumeConsumption(int channelIndex) |
void |
retriggerPartitionRequest(IntermediateResultPartitionID partitionId)
Retriggers a partition request.
|
void |
sendTaskEvent(TaskEvent event) |
void |
setBufferPool(BufferPool bufferPool) |
void |
setInputChannels(InputChannel... channels) |
void |
setup()
Setup gate, potentially heavy-weight, blocking operation comparing to just creation.
|
void |
updateInputChannel(ResourceID localLocation,
NettyShuffleDescriptor shuffleDescriptor) |
getAvailableFuture, getChannelInfos
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
isApproximatelyAvailable, isAvailable
public SingleInputGate(String owningTaskName, int gateIndex, IntermediateDataSetID consumedResultId, ResultPartitionType consumedPartitionType, int consumedSubpartitionIndex, int numberOfInputChannels, PartitionProducerStateProvider partitionProducerStateProvider, SupplierWithException<BufferPool,IOException> bufferPoolFactory, @Nullable BufferDecompressor bufferDecompressor, MemorySegmentProvider memorySegmentProvider)
public void setup() throws IOException
InputGate
setup
in class InputGate
IOException
public CompletableFuture<?> readRecoveredState(ExecutorService executor, ChannelStateReader reader)
InputGate
readRecoveredState
in class InputGate
executor
- the dedicated executor for performing this action for all the internal
channels.reader
- the dedicated reader for unspilling the respective channel state from
snapshots.public void requestPartitions()
requestPartitions
in class InputGate
public void registerBufferReceivedListener(BufferReceivedListener bufferReceivedListener)
registerBufferReceivedListener
in class InputGate
public int getNumberOfInputChannels()
getNumberOfInputChannels
in class InputGate
public int getGateIndex()
IndexedInputGate
getGateIndex
in class IndexedInputGate
public ResultPartitionType getConsumedPartitionType()
public BufferPool getBufferPool()
public String getOwningTaskName()
public int getNumberOfQueuedBuffers()
public CompletableFuture<Void> getCloseFuture()
public InputChannel getChannel(int channelIndex)
InputGate
getChannel
in class InputGate
public void setBufferPool(BufferPool bufferPool)
@VisibleForTesting public void assignExclusiveSegments() throws IOException
IOException
public void setInputChannels(InputChannel... channels)
public void updateInputChannel(ResourceID localLocation, NettyShuffleDescriptor shuffleDescriptor) throws IOException, InterruptedException
IOException
InterruptedException
public void retriggerPartitionRequest(IntermediateResultPartitionID partitionId) throws IOException
IOException
public void close() throws IOException
IOException
public boolean isFinished()
isFinished
in interface PullingAsyncDataInput<BufferOrEvent>
isFinished
in class InputGate
public Optional<BufferOrEvent> getNext() throws IOException, InterruptedException
InputGate
BufferOrEvent
.
Note: It should be guaranteed that the previous returned buffer has been recycled before getting next one.
getNext
in class InputGate
Optional.empty()
if InputGate.isFinished()
returns true.IOException
InterruptedException
public Optional<BufferOrEvent> pollNext() throws IOException, InterruptedException
InputGate
BufferOrEvent
.
Note: It should be guaranteed that the previous returned buffer has been recycled before polling next one.
pollNext
in interface PullingAsyncDataInput<BufferOrEvent>
pollNext
in class InputGate
Optional.empty()
if there is no data to return or if InputGate.isFinished()
returns true.IOException
InterruptedException
public void sendTaskEvent(TaskEvent event) throws IOException
sendTaskEvent
in class InputGate
IOException
public void resumeConsumption(int channelIndex) throws IOException
resumeConsumption
in class InputGate
IOException
public Map<IntermediateResultPartitionID,InputChannel> getInputChannels()
Copyright © 2014–2021 The Apache Software Foundation. All rights reserved.