public class SingleInputGate extends IndexedInputGate
Each intermediate result is partitioned over its producing parallel subtasks; each of these partitions is furthermore partitioned into one or more subpartitions.
As an example, consider a map-reduce program, where the map operator produces data and the reduce operator consumes the produced data.
+-----+ +---------------------+ +--------+
| Map | = produce => | Intermediate Result | <= consume = | Reduce |
+-----+ +---------------------+ +--------+
When deploying such a program in parallel, the intermediate result will be partitioned over its producing parallel subtasks; each of these partitions is furthermore partitioned into one or more subpartitions.
Intermediate result
+-----------------------------------------+
| +----------------+ | +-----------------------+
+-------+ | +-------------+ +=> | Subpartition 1 | | <=======+=== | Input Gate | Reduce 1 |
| Map 1 | ==> | | Partition 1 | =| +----------------+ | | +-----------------------+
+-------+ | +-------------+ +=> | Subpartition 2 | | <==+ |
| +----------------+ | | | Subpartition request
| | | |
| +----------------+ | | |
+-------+ | +-------------+ +=> | Subpartition 1 | | <==+====+
| Map 2 | ==> | | Partition 2 | =| +----------------+ | | +-----------------------+
+-------+ | +-------------+ +=> | Subpartition 2 | | <==+======== | Input Gate | Reduce 2 |
| +----------------+ | +-----------------------+
+-----------------------------------------+
In the above example, two map subtasks produce the intermediate result in parallel, resulting in two partitions (Partition 1 and 2). Each of these partitions is further partitioned into two subpartitions -- one for each parallel reduce subtask.
InputGate.InputWithData<INPUT,DATA>PullingAsyncDataInput.EndOfDataStatusAvailabilityProvider.AvailabilityHelperavailabilityHelper, priorityAvailabilityHelperAVAILABLE| Constructor and Description |
|---|
SingleInputGate(String owningTaskName,
int gateIndex,
IntermediateDataSetID consumedResultId,
ResultPartitionType consumedPartitionType,
int numberOfInputChannels,
PartitionProducerStateProvider partitionProducerStateProvider,
SupplierWithException<BufferPool,IOException> bufferPoolFactory,
BufferDecompressor bufferDecompressor,
MemorySegmentProvider memorySegmentProvider,
int segmentSize,
ThroughputCalculator throughputCalculator,
BufferDebloater bufferDebloater) |
blockConsumption, checkpointStarted, checkpointStopped, convertToPriorityEvent, getInputGateIndexgetAvailableFuture, getChannelInfos, getPriorityEventAvailableFuture, setChannelStateWriterclone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, waitgetChannelInfosand, isApproximatelyAvailable, isAvailable, orpublic SingleInputGate(String owningTaskName, int gateIndex, IntermediateDataSetID consumedResultId, ResultPartitionType consumedPartitionType, int numberOfInputChannels, PartitionProducerStateProvider partitionProducerStateProvider, SupplierWithException<BufferPool,IOException> bufferPoolFactory, @Nullable BufferDecompressor bufferDecompressor, MemorySegmentProvider memorySegmentProvider, int segmentSize, ThroughputCalculator throughputCalculator, @Nullable BufferDebloater bufferDebloater)
protected PrioritizedDeque<InputChannel> getInputChannelsWithData()
public void setup()
throws IOException
InputGatesetup in class InputGateIOExceptionpublic CompletableFuture<Void> getStateConsumedFuture()
getStateConsumedFuture in class InputGatepublic void requestPartitions()
requestPartitions in class InputGate@VisibleForTesting public void convertRecoveredInputChannels()
public void finishReadRecoveredState()
throws IOException
finishReadRecoveredState in class InputGateIOExceptionpublic int getNumberOfInputChannels()
getNumberOfInputChannels in interface CheckpointableInputgetNumberOfInputChannels in class InputGatepublic int getGateIndex()
IndexedInputGategetGateIndex in class IndexedInputGatepublic List<InputChannelInfo> getUnfinishedChannels()
IndexedInputGategetUnfinishedChannels in class IndexedInputGate@VisibleForTesting public void announceBufferSize(int newBufferSize)
public void triggerDebloating()
triggerDebloating in class IndexedInputGatepublic Duration getLastEstimatedTimeToConsume()
public ResultPartitionType getConsumedPartitionType()
public BufferPool getBufferPool()
public String getOwningTaskName()
public int getNumberOfQueuedBuffers()
public long getSizeOfQueuedBuffers()
public CompletableFuture<Void> getCloseFuture()
public InputChannel getChannel(int channelIndex)
InputGategetChannel in class InputGatepublic void setBufferPool(BufferPool bufferPool)
@VisibleForTesting public void setupChannels() throws IOException
IOExceptionpublic void setInputChannels(InputChannel... channels)
public void setTieredStorageService(List<TieredStorageConsumerSpec> tieredStorageConsumerSpecs, TieredStorageConsumerClient client, TieredStorageNettyServiceImpl nettyService)
public void updateInputChannel(ResourceID localLocation, NettyShuffleDescriptor shuffleDescriptor) throws IOException, InterruptedException
IOExceptionInterruptedExceptionpublic void retriggerPartitionRequest(IntermediateResultPartitionID partitionId, InputChannelInfo inputChannelInfo) throws IOException
IOExceptionpublic void close()
throws IOException
IOExceptionpublic boolean isFinished()
isFinished in interface PullingAsyncDataInput<BufferOrEvent>isFinished in class InputGatepublic PullingAsyncDataInput.EndOfDataStatus hasReceivedEndOfData()
PullingAsyncDataInputMoreover it tells us the reason why there is no more data incoming. If any of the upstream
subtasks finished because of the stop-with-savepoint --no-drain, we should not drain the
input. See also StopMode.
public Optional<BufferOrEvent> getNext() throws IOException, InterruptedException
InputGateBufferOrEvent.
Note: It should be guaranteed that the previous returned buffer has been recycled before getting next one.
getNext in class InputGateOptional.empty() if InputGate.isFinished() returns true.IOExceptionInterruptedExceptionpublic Optional<BufferOrEvent> pollNext() throws IOException, InterruptedException
InputGateBufferOrEvent.
Note: It should be guaranteed that the previous returned buffer has been recycled before polling next one.
pollNext in interface PullingAsyncDataInput<BufferOrEvent>pollNext in class InputGateOptional.empty() if there is no data to return or if InputGate.isFinished()
returns true.IOExceptionInterruptedExceptionpublic void sendTaskEvent(TaskEvent event) throws IOException
sendTaskEvent in class InputGateIOExceptionpublic void resumeConsumption(InputChannelInfo channelInfo) throws IOException
resumeConsumption in interface CheckpointableInputresumeConsumption in class InputGateIOExceptionpublic void acknowledgeAllRecordsProcessed(InputChannelInfo channelInfo) throws IOException
acknowledgeAllRecordsProcessed in class InputGateIOException@VisibleForTesting public Map<Tuple2<IntermediateResultPartitionID,InputChannelInfo>,InputChannel> getInputChannels()
public Iterable<InputChannel> inputChannels()
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.