public interface InputGate
Each intermediate result is partitioned over its producing parallel subtasks; each of these partitions is furthermore partitioned into one or more subpartitions.
As an example, consider a map-reduce program, where the map operator produces data and the reduce operator consumes the produced data.
+-----+ +---------------------+ +--------+
| Map | = produce => | Intermediate Result | <= consume = | Reduce |
+-----+ +---------------------+ +--------+
When deploying such a program in parallel, the intermediate result will be partitioned over its producing parallel subtasks; each of these partitions is furthermore partitioned into one or more subpartitions.
Intermediate result
+-----------------------------------------+
| +----------------+ | +-----------------------+
+-------+ | +-------------+ +=> | Subpartition 1 | | <=======+=== | Input Gate | Reduce 1 |
| Map 1 | ==> | | Partition 1 | =| +----------------+ | | +-----------------------+
+-------+ | +-------------+ +=> | Subpartition 2 | | <==+ |
| +----------------+ | | | Subpartition request
| | | |
| +----------------+ | | |
+-------+ | +-------------+ +=> | Subpartition 1 | | <==+====+
| Map 2 | ==> | | Partition 2 | =| +----------------+ | | +-----------------------+
+-------+ | +-------------+ +=> | Subpartition 2 | | <==+======== | Input Gate | Reduce 2 |
| +----------------+ | +-----------------------+
+-----------------------------------------+
In the above example, two map subtasks produce the intermediate result in parallel, resulting in two partitions (Partition 1 and 2). Each of these partitions is further partitioned into two subpartitions -- one for each parallel reduce subtask.
Modifier and Type | Method and Description |
---|---|
BufferOrEvent |
getNextBufferOrEvent() |
int |
getNumberOfInputChannels() |
int |
getPageSize() |
boolean |
isFinished() |
void |
registerListener(EventListener<InputGate> listener) |
void |
requestPartitions() |
void |
sendTaskEvent(TaskEvent event) |
int getNumberOfInputChannels()
boolean isFinished()
void requestPartitions() throws IOException, InterruptedException
IOException
InterruptedException
BufferOrEvent getNextBufferOrEvent() throws IOException, InterruptedException
IOException
InterruptedException
void sendTaskEvent(TaskEvent event) throws IOException
IOException
void registerListener(EventListener<InputGate> listener)
int getPageSize()
Copyright © 2014–2017 The Apache Software Foundation. All rights reserved.