org.apache.flink.runtime.io.network.partition.consumer

Class SingleInputGate

java.lang.Object

org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate

All Implemented Interfaces:

InputGate

public class SingleInputGate
extends Object
implements InputGate

An input gate consumes one or more partitions of a single produced intermediate result.

Each intermediate result is partitioned over its producing parallel subtasks; each of these partitions is furthermore partitioned into one or more subpartitions.

As an example, consider a map-reduce program, where the map operator produces data and the reduce operator consumes the produced data.

 +-----+              +---------------------+              +--------+
 | Map | = produce => | Intermediate Result | <= consume = | Reduce |
 +-----+              +---------------------+              +--------+
 

When deploying such a program in parallel, the intermediate result will be partitioned over its producing parallel subtasks; each of these partitions is furthermore partitioned into one or more subpartitions.

                            Intermediate result
               +-----------------------------------------+
               |                      +----------------+ |              +-----------------------+
 +-------+     | +-------------+  +=> | Subpartition 1 | | <=======+=== | Input Gate | Reduce 1 |
 | Map 1 | ==> | | Partition 1 | =|   +----------------+ |         |    +-----------------------+
 +-------+     | +-------------+  +=> | Subpartition 2 | | <==+    |
               |                      +----------------+ |    |    | Subpartition request
               |                                         |    |    |
               |                      +----------------+ |    |    |
 +-------+     | +-------------+  +=> | Subpartition 1 | | <==+====+
 | Map 2 | ==> | | Partition 2 | =|   +----------------+ |    |         +-----------------------+
 +-------+     | +-------------+  +=> | Subpartition 2 | | <==+======== | Input Gate | Reduce 2 |
               |                      +----------------+ |              +-----------------------+
               +-----------------------------------------+
 

In the above example, two map subtasks produce the intermediate result in parallel, resulting in two partitions (Partition 1 and 2). Each of these partitions is further partitioned into two subpartitions -- one for each parallel reduce subtask.

Constructor Summary

Constructors

Constructor and Description
SingleInputGate(String owningTaskName, JobID jobId, IntermediateDataSetID consumedResultId, ResultPartitionType consumedPartitionType, int consumedSubpartitionIndex, int numberOfInputChannels, TaskActions taskActions, TaskIOMetricGroup metrics, boolean isCreditBased)

Method Summary

Modifier and Type	Method and Description
void	assignExclusiveSegments(NetworkBufferPool networkBufferPool, int networkBuffersPerChannel) Assign the exclusive buffers to all remote input channels directly for credit-based mode.
static SingleInputGate	create(String owningTaskName, JobID jobId, ExecutionAttemptID executionId, InputGateDeploymentDescriptor igdd, NetworkEnvironment networkEnvironment, TaskActions taskActions, TaskIOMetricGroup metrics) Creates an input gate and all of its input channels.
BufferPool	getBufferPool()
ResultPartitionType	getConsumedPartitionType() Returns the type of this input channel's consumed result partition.
IntermediateDataSetID	getConsumedResultId()
Optional<BufferOrEvent>	getNextBufferOrEvent() Blocking call waiting for next BufferOrEvent.
int	getNumberOfInputChannels()
int	getNumberOfQueuedBuffers()
String	getOwningTaskName()
int	getPageSize()
boolean	isFinished()
Optional<BufferOrEvent>	pollNextBufferOrEvent() Poll the BufferOrEvent.
void	registerListener(InputGateListener inputGateListener)
void	releaseAllResources()
void	requestPartitions()
void	retriggerPartitionRequest(IntermediateResultPartitionID partitionId) Retriggers a partition request.
void	returnExclusiveSegments(List<MemorySegment> segments) The exclusive segments are recycled to network buffer pool directly when input channel is released.
void	sendTaskEvent(TaskEvent event)
void	setBufferPool(BufferPool bufferPool)
void	setInputChannel(IntermediateResultPartitionID partitionId, InputChannel inputChannel)
void	updateInputChannel(InputChannelDeploymentDescriptor icdd)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SingleInputGate

public SingleInputGate(String owningTaskName,
                       JobID jobId,
                       IntermediateDataSetID consumedResultId,
                       ResultPartitionType consumedPartitionType,
                       int consumedSubpartitionIndex,
                       int numberOfInputChannels,
                       TaskActions taskActions,
                       TaskIOMetricGroup metrics,
                       boolean isCreditBased)

Method Detail

getNumberOfInputChannels

public int getNumberOfInputChannels()

Specified by:

getNumberOfInputChannels in interface InputGate

getConsumedResultId

public IntermediateDataSetID getConsumedResultId()

getConsumedPartitionType

public ResultPartitionType getConsumedPartitionType()

Returns the type of this input channel's consumed result partition.

Returns:

consumed result partition type

getBufferPool

public BufferPool getBufferPool()

getPageSize

public int getPageSize()

Specified by:

getPageSize in interface InputGate

getNumberOfQueuedBuffers

public int getNumberOfQueuedBuffers()

getOwningTaskName

public String getOwningTaskName()

Specified by:

getOwningTaskName in interface InputGate

setBufferPool

public void setBufferPool(BufferPool bufferPool)

assignExclusiveSegments

public void assignExclusiveSegments(NetworkBufferPool networkBufferPool,
                                    int networkBuffersPerChannel)
                             throws IOException

Assign the exclusive buffers to all remote input channels directly for credit-based mode.

Parameters:

networkBufferPool - The global pool to request and recycle exclusive buffers

networkBuffersPerChannel - The number of exclusive buffers for each channel

Throws:

IOException

returnExclusiveSegments

public void returnExclusiveSegments(List<MemorySegment> segments)
                             throws IOException

The exclusive segments are recycled to network buffer pool directly when input channel is released.

Parameters:

segments - The exclusive segments need to be recycled

Throws:

IOException

setInputChannel

public void setInputChannel(IntermediateResultPartitionID partitionId,
                            InputChannel inputChannel)

updateInputChannel

public void updateInputChannel(InputChannelDeploymentDescriptor icdd)
                        throws IOException,
                               InterruptedException

Throws:

IOException

InterruptedException

retriggerPartitionRequest

public void retriggerPartitionRequest(IntermediateResultPartitionID partitionId)
                               throws IOException,
                                      InterruptedException

Retriggers a partition request.

Throws:

IOException

InterruptedException

releaseAllResources

public void releaseAllResources()
                         throws IOException

Throws:

IOException

isFinished

public boolean isFinished()

Specified by:

isFinished in interface InputGate

requestPartitions

public void requestPartitions()
                       throws IOException,
                              InterruptedException

Specified by:

requestPartitions in interface InputGate

Throws:

IOException

InterruptedException

getNextBufferOrEvent

public Optional<BufferOrEvent> getNextBufferOrEvent()
                                             throws IOException,
                                                    InterruptedException

Description copied from interface: InputGate

Blocking call waiting for next BufferOrEvent.

Specified by:

getNextBufferOrEvent in interface InputGate

Returns:

Optional.empty() if InputGate.isFinished() returns true.

Throws:

IOException

InterruptedException

pollNextBufferOrEvent

public Optional<BufferOrEvent> pollNextBufferOrEvent()
                                              throws IOException,
                                                     InterruptedException

Description copied from interface: InputGate

Poll the BufferOrEvent.

Specified by:

pollNextBufferOrEvent in interface InputGate

Returns:

Optional.empty() if there is no data to return or if InputGate.isFinished() returns true.

Throws:

IOException

InterruptedException

sendTaskEvent

public void sendTaskEvent(TaskEvent event)
                   throws IOException

Specified by:

sendTaskEvent in interface InputGate

Throws:

IOException

registerListener

public void registerListener(InputGateListener inputGateListener)

Specified by:

registerListener in interface InputGate

create

public static SingleInputGate create(String owningTaskName,
                                     JobID jobId,
                                     ExecutionAttemptID executionId,
                                     InputGateDeploymentDescriptor igdd,
                                     NetworkEnvironment networkEnvironment,
                                     TaskActions taskActions,
                                     TaskIOMetricGroup metrics)

Creates an input gate and all of its input channels.