Package | Description |
---|---|
org.apache.flink.runtime.iterative.task | |
org.apache.flink.runtime.jobgraph | |
org.apache.flink.runtime.operators | |
org.apache.flink.runtime.operators.chaining | |
org.apache.flink.runtime.operators.hash | |
org.apache.flink.runtime.operators.resettable | |
org.apache.flink.runtime.operators.sort | |
org.apache.flink.streaming.api.graph | |
org.apache.flink.streaming.runtime.io | |
org.apache.flink.streaming.runtime.tasks |
This package contains classes that realize streaming tasks.
|
Modifier and Type | Class and Description |
---|---|
class |
AbstractIterativeTask<S extends Function,OT>
The abstract base class for all tasks able to participate in an iteration.
|
class |
IterationHeadTask<X,Y,S extends Function,OT>
The head is responsible for coordinating an iteration and can run a
Driver inside. |
class |
IterationIntermediateTask<S extends Function,OT>
An intermediate iteration task, which runs a
Driver
inside. |
class |
IterationSynchronizationSinkTask
The task responsible for synchronizing all iteration heads, implemented as an output task.
|
class |
IterationTailTask<S extends Function,OT>
An iteration tail, which runs a driver inside.
|
Modifier and Type | Method and Description |
---|---|
Class<? extends AbstractInvokable> |
JobVertex.getInvokableClass(ClassLoader cl)
Returns the invokable class which represents the task of this vertex.
|
Modifier and Type | Method and Description |
---|---|
void |
JobVertex.setInvokableClass(Class<? extends AbstractInvokable> invokable) |
Modifier and Type | Class and Description |
---|---|
class |
BatchTask<S extends Function,OT>
The base class for all batch tasks.
|
class |
DataSinkTask<IT>
DataSinkTask which is executed by a task manager.
|
class |
DataSourceTask<OT>
DataSourceTask which is executed by a task manager.
|
Modifier and Type | Method and Description |
---|---|
AbstractInvokable |
TaskContext.getContainingTask() |
AbstractInvokable |
BatchTask.getContainingTask() |
Modifier and Type | Method and Description |
---|---|
static void |
BatchTask.closeChainedTasks(List<ChainedDriver<?,?>> tasks,
AbstractInvokable parent)
Closes all chained tasks, in the order as they are stored in the array.
|
static String |
BatchTask.constructLogString(String message,
String taskName,
AbstractInvokable parent)
Utility function that composes a string for logging purposes.
|
static <T> Collector<T> |
BatchTask.getOutputCollector(AbstractInvokable task,
TaskConfig config,
ClassLoader cl,
List<RecordWriter<?>> eventualOutputs,
int outputOffset,
int numOutputs)
Creates the
Collector for the given task, as described by the given configuration. |
static <T> Collector<T> |
BatchTask.initOutputs(AbstractInvokable containingTask,
ClassLoader cl,
TaskConfig config,
List<ChainedDriver<?,?>> chainedTasksTarget,
List<RecordWriter<?>> eventualOutputs,
ExecutionConfig executionConfig,
Map<String,Accumulator<?,?>> accumulatorMap)
Creates a writer for each output.
|
static void |
BatchTask.logAndThrowException(Exception ex,
AbstractInvokable parent)
Prints an error message and throws the given exception.
|
static void |
BatchTask.openChainedTasks(List<ChainedDriver<?,?>> tasks,
AbstractInvokable parent)
Opens all chained tasks, in the order as they are stored in the array.
|
void |
NoOpChainedDriver.setup(AbstractInvokable parent) |
Constructor and Description |
---|
TempBarrier(AbstractInvokable owner,
MutableObjectIterator<T> input,
TypeSerializerFactory<T> serializerFactory,
MemoryManager memManager,
IOManager ioManager,
int numPages,
List<MemorySegment> preAllocated) |
Modifier and Type | Method and Description |
---|---|
void |
ChainedFlatMapDriver.setup(AbstractInvokable parent) |
void |
ChainedReduceCombineDriver.setup(AbstractInvokable parent) |
abstract void |
ChainedDriver.setup(AbstractInvokable parent) |
void |
ChainedAllReduceDriver.setup(AbstractInvokable parent) |
void |
ChainedMapDriver.setup(AbstractInvokable parent) |
void |
ChainedTerminationCriterionDriver.setup(AbstractInvokable parent) |
void |
SynchronousChainedCombineDriver.setup(AbstractInvokable parent) |
void |
ChainedDriver.setup(TaskConfig config,
String taskName,
Collector<OT> outputCollector,
AbstractInvokable parent,
ClassLoader userCodeClassLoader,
ExecutionConfig executionConfig,
Map<String,Accumulator<?,?>> accumulatorMap) |
Modifier and Type | Method and Description |
---|---|
<BT,PT> MutableHashTable<BT,PT> |
ReusingBuildFirstReOpenableHashJoinIterator.getHashJoin(TypeSerializer<BT> buildSideSerializer,
TypeComparator<BT> buildSideComparator,
TypeSerializer<PT> probeSideSerializer,
TypeComparator<PT> probeSideComparator,
TypePairComparator<PT,BT> pairComparator,
MemoryManager memManager,
IOManager ioManager,
AbstractInvokable ownerTask,
double memoryFraction,
boolean useBitmapFilters) |
<BT,PT> MutableHashTable<BT,PT> |
NonReusingBuildFirstReOpenableHashJoinIterator.getHashJoin(TypeSerializer<BT> buildSideSerializer,
TypeComparator<BT> buildSideComparator,
TypeSerializer<PT> probeSideSerializer,
TypeComparator<PT> probeSideComparator,
TypePairComparator<PT,BT> pairComparator,
MemoryManager memManager,
IOManager ioManager,
AbstractInvokable ownerTask,
double memoryFraction,
boolean useBitmapFilters) |
<BT,PT> MutableHashTable<BT,PT> |
HashJoinIteratorBase.getHashJoin(TypeSerializer<BT> buildSideSerializer,
TypeComparator<BT> buildSideComparator,
TypeSerializer<PT> probeSideSerializer,
TypeComparator<PT> probeSideComparator,
TypePairComparator<PT,BT> pairComparator,
MemoryManager memManager,
IOManager ioManager,
AbstractInvokable ownerTask,
double memoryFraction,
boolean useBloomFilters) |
<BT,PT> MutableHashTable<BT,PT> |
ReusingBuildSecondReOpenableHashJoinIterator.getHashJoin(TypeSerializer<BT> buildSideSerializer,
TypeComparator<BT> buildSideComparator,
TypeSerializer<PT> probeSideSerializer,
TypeComparator<PT> probeSideComparator,
TypePairComparator<PT,BT> pairComparator,
MemoryManager memManager,
IOManager ioManager,
AbstractInvokable ownerTask,
double memoryFraction,
boolean useBitmapFilters) |
<BT,PT> MutableHashTable<BT,PT> |
NonReusingBuildSecondReOpenableHashJoinIterator.getHashJoin(TypeSerializer<BT> buildSideSerializer,
TypeComparator<BT> buildSideComparator,
TypeSerializer<PT> probeSideSerializer,
TypeComparator<PT> probeSideComparator,
TypePairComparator<PT,BT> pairComparator,
MemoryManager memManager,
IOManager ioManager,
AbstractInvokable ownerTask,
double memoryFraction,
boolean useBitmapFilters) |
Constructor and Description |
---|
NonReusingBuildFirstHashJoinIterator(MutableObjectIterator<V1> firstInput,
MutableObjectIterator<V2> secondInput,
TypeSerializer<V1> serializer1,
TypeComparator<V1> comparator1,
TypeSerializer<V2> serializer2,
TypeComparator<V2> comparator2,
TypePairComparator<V2,V1> pairComparator,
MemoryManager memManager,
IOManager ioManager,
AbstractInvokable ownerTask,
double memoryFraction,
boolean probeSideOuterJoin,
boolean buildSideOuterJoin,
boolean useBitmapFilters) |
NonReusingBuildFirstReOpenableHashJoinIterator(MutableObjectIterator<V1> firstInput,
MutableObjectIterator<V2> secondInput,
TypeSerializer<V1> serializer1,
TypeComparator<V1> comparator1,
TypeSerializer<V2> serializer2,
TypeComparator<V2> comparator2,
TypePairComparator<V2,V1> pairComparator,
MemoryManager memManager,
IOManager ioManager,
AbstractInvokable ownerTask,
double memoryFraction,
boolean probeSideOuterJoin,
boolean buildSideOuterJoin,
boolean useBitmapFilters) |
NonReusingBuildSecondHashJoinIterator(MutableObjectIterator<V1> firstInput,
MutableObjectIterator<V2> secondInput,
TypeSerializer<V1> serializer1,
TypeComparator<V1> comparator1,
TypeSerializer<V2> serializer2,
TypeComparator<V2> comparator2,
TypePairComparator<V1,V2> pairComparator,
MemoryManager memManager,
IOManager ioManager,
AbstractInvokable ownerTask,
double memoryFraction,
boolean probeSideOuterJoin,
boolean buildSideOuterJoin,
boolean useBitmapFilters) |
NonReusingBuildSecondReOpenableHashJoinIterator(MutableObjectIterator<V1> firstInput,
MutableObjectIterator<V2> secondInput,
TypeSerializer<V1> serializer1,
TypeComparator<V1> comparator1,
TypeSerializer<V2> serializer2,
TypeComparator<V2> comparator2,
TypePairComparator<V1,V2> pairComparator,
MemoryManager memManager,
IOManager ioManager,
AbstractInvokable ownerTask,
double memoryFraction,
boolean probeSideOuterJoin,
boolean buildSideOuterJoin,
boolean useBitmapFilters) |
ReusingBuildFirstHashJoinIterator(MutableObjectIterator<V1> firstInput,
MutableObjectIterator<V2> secondInput,
TypeSerializer<V1> serializer1,
TypeComparator<V1> comparator1,
TypeSerializer<V2> serializer2,
TypeComparator<V2> comparator2,
TypePairComparator<V2,V1> pairComparator,
MemoryManager memManager,
IOManager ioManager,
AbstractInvokable ownerTask,
double memoryFraction,
boolean probeSideOuterJoin,
boolean buildSideOuterJoin,
boolean useBitmapFilters) |
ReusingBuildFirstReOpenableHashJoinIterator(MutableObjectIterator<V1> firstInput,
MutableObjectIterator<V2> secondInput,
TypeSerializer<V1> serializer1,
TypeComparator<V1> comparator1,
TypeSerializer<V2> serializer2,
TypeComparator<V2> comparator2,
TypePairComparator<V2,V1> pairComparator,
MemoryManager memManager,
IOManager ioManager,
AbstractInvokable ownerTask,
double memoryFraction,
boolean probeSideOuterJoin,
boolean buildSideOuterJoin,
boolean useBitmapFilters) |
ReusingBuildSecondHashJoinIterator(MutableObjectIterator<V1> firstInput,
MutableObjectIterator<V2> secondInput,
TypeSerializer<V1> serializer1,
TypeComparator<V1> comparator1,
TypeSerializer<V2> serializer2,
TypeComparator<V2> comparator2,
TypePairComparator<V1,V2> pairComparator,
MemoryManager memManager,
IOManager ioManager,
AbstractInvokable ownerTask,
double memoryFraction,
boolean probeSideOuterJoin,
boolean buildSideOuterJoin,
boolean useBitmapFilters) |
ReusingBuildSecondReOpenableHashJoinIterator(MutableObjectIterator<V1> firstInput,
MutableObjectIterator<V2> secondInput,
TypeSerializer<V1> serializer1,
TypeComparator<V1> comparator1,
TypeSerializer<V2> serializer2,
TypeComparator<V2> comparator2,
TypePairComparator<V1,V2> pairComparator,
MemoryManager memManager,
IOManager ioManager,
AbstractInvokable ownerTask,
double memoryFraction,
boolean probeSideOuterJoin,
boolean buildSideOuterJoin,
boolean useBitmapFilters) |
Constructor and Description |
---|
AbstractMergeInnerJoinIterator(MutableObjectIterator<T1> input1,
MutableObjectIterator<T2> input2,
TypeSerializer<T1> serializer1,
TypeComparator<T1> comparator1,
TypeSerializer<T2> serializer2,
TypeComparator<T2> comparator2,
TypePairComparator<T1,T2> pairComparator,
MemoryManager memoryManager,
IOManager ioManager,
int numMemoryPages,
AbstractInvokable parentTask) |
AbstractMergeIterator(MutableObjectIterator<T1> input1,
MutableObjectIterator<T2> input2,
TypeSerializer<T1> serializer1,
TypeComparator<T1> comparator1,
TypeSerializer<T2> serializer2,
TypeComparator<T2> comparator2,
TypePairComparator<T1,T2> pairComparator,
MemoryManager memoryManager,
IOManager ioManager,
int numMemoryPages,
AbstractInvokable parentTask) |
AbstractMergeOuterJoinIterator(OuterJoinOperatorBase.OuterJoinType outerJoinType,
MutableObjectIterator<T1> input1,
MutableObjectIterator<T2> input2,
TypeSerializer<T1> serializer1,
TypeComparator<T1> comparator1,
TypeSerializer<T2> serializer2,
TypeComparator<T2> comparator2,
TypePairComparator<T1,T2> pairComparator,
MemoryManager memoryManager,
IOManager ioManager,
int numMemoryPages,
AbstractInvokable parentTask) |
CombiningSpillingThread(ExceptionHandler<IOException> exceptionHandler,
UnilateralSortMerger.CircularQueues<E> queues,
AbstractInvokable parentTask,
MemoryManager memManager,
IOManager ioManager,
TypeSerializer<E> serializer,
TypeComparator<E> comparator,
List<MemorySegment> sortReadMemory,
List<MemorySegment> writeMemory,
int maxNumFileHandles,
boolean objectReuseEnabled) |
CombiningUnilateralSortMerger(GroupCombineFunction<E,E> combineStub,
MemoryManager memoryManager,
IOManager ioManager,
MutableObjectIterator<E> input,
AbstractInvokable parentTask,
TypeSerializerFactory<E> serializerFactory,
TypeComparator<E> comparator,
double memoryFraction,
int maxNumFileHandles,
float startSpillingFraction,
boolean handleLargeRecords,
boolean objectReuseEnabled)
Creates a new sorter that reads the data from a given reader and provides an iterator
returning that data in a sorted manner.
|
CombiningUnilateralSortMerger(GroupCombineFunction<E,E> combineStub,
MemoryManager memoryManager,
IOManager ioManager,
MutableObjectIterator<E> input,
AbstractInvokable parentTask,
TypeSerializerFactory<E> serializerFactory,
TypeComparator<E> comparator,
double memoryFraction,
int numSortBuffers,
int maxNumFileHandles,
float startSpillingFraction,
boolean handleLargeRecords,
boolean objectReuseEnabled)
Creates a new sorter that reads the data from a given reader and provides an iterator
returning that data in a sorted manner.
|
LargeRecordHandler(TypeSerializer<T> serializer,
TypeComparator<T> comparator,
IOManager ioManager,
MemoryManager memManager,
List<MemorySegment> memory,
AbstractInvokable memoryOwner,
int maxFilehandles) |
NonReusingMergeInnerJoinIterator(MutableObjectIterator<T1> input1,
MutableObjectIterator<T2> input2,
TypeSerializer<T1> serializer1,
TypeComparator<T1> comparator1,
TypeSerializer<T2> serializer2,
TypeComparator<T2> comparator2,
TypePairComparator<T1,T2> pairComparator,
MemoryManager memoryManager,
IOManager ioManager,
int numMemoryPages,
AbstractInvokable parentTask) |
NonReusingMergeOuterJoinIterator(OuterJoinOperatorBase.OuterJoinType outerJoinType,
MutableObjectIterator<T1> input1,
MutableObjectIterator<T2> input2,
TypeSerializer<T1> serializer1,
TypeComparator<T1> comparator1,
TypeSerializer<T2> serializer2,
TypeComparator<T2> comparator2,
TypePairComparator<T1,T2> pairComparator,
MemoryManager memoryManager,
IOManager ioManager,
int numMemoryPages,
AbstractInvokable parentTask) |
ReadingThread(ExceptionHandler<IOException> exceptionHandler,
MutableObjectIterator<E> reader,
UnilateralSortMerger.CircularQueues<E> queues,
LargeRecordHandler<E> largeRecordsHandler,
E readTarget,
AbstractInvokable parentTask,
long startSpillingBytes)
Creates a new reading thread.
|
ReusingMergeInnerJoinIterator(MutableObjectIterator<T1> input1,
MutableObjectIterator<T2> input2,
TypeSerializer<T1> serializer1,
TypeComparator<T1> comparator1,
TypeSerializer<T2> serializer2,
TypeComparator<T2> comparator2,
TypePairComparator<T1,T2> pairComparator,
MemoryManager memoryManager,
IOManager ioManager,
int numMemoryPages,
AbstractInvokable parentTask) |
ReusingMergeOuterJoinIterator(OuterJoinOperatorBase.OuterJoinType outerJoinType,
MutableObjectIterator<T1> input1,
MutableObjectIterator<T2> input2,
TypeSerializer<T1> serializer1,
TypeComparator<T1> comparator1,
TypeSerializer<T2> serializer2,
TypeComparator<T2> comparator2,
TypePairComparator<T1,T2> pairComparator,
MemoryManager memoryManager,
IOManager ioManager,
int numMemoryPages,
AbstractInvokable parentTask) |
SortingThread(ExceptionHandler<IOException> exceptionHandler,
UnilateralSortMerger.CircularQueues<E> queues,
AbstractInvokable parentTask)
Creates a new sorting thread.
|
SpillingThread(ExceptionHandler<IOException> exceptionHandler,
UnilateralSortMerger.CircularQueues<E> queues,
AbstractInvokable parentTask,
MemoryManager memManager,
IOManager ioManager,
TypeSerializer<E> serializer,
TypeComparator<E> comparator,
List<MemorySegment> sortReadMemory,
List<MemorySegment> writeMemory,
int maxNumFileHandles)
Creates the spilling thread.
|
ThreadBase(ExceptionHandler<IOException> exceptionHandler,
String name,
UnilateralSortMerger.CircularQueues<E> queues,
AbstractInvokable parentTask)
Creates a new thread.
|
UnilateralSortMerger(MemoryManager memoryManager,
IOManager ioManager,
MutableObjectIterator<E> input,
AbstractInvokable parentTask,
TypeSerializerFactory<E> serializerFactory,
TypeComparator<E> comparator,
double memoryFraction,
int maxNumFileHandles,
float startSpillingFraction,
boolean handleLargeRecords,
boolean objectReuseEnabled) |
UnilateralSortMerger(MemoryManager memoryManager,
IOManager ioManager,
MutableObjectIterator<E> input,
AbstractInvokable parentTask,
TypeSerializerFactory<E> serializerFactory,
TypeComparator<E> comparator,
double memoryFraction,
int numSortBuffers,
int maxNumFileHandles,
float startSpillingFraction,
boolean handleLargeRecords,
boolean objectReuseEnabled) |
UnilateralSortMerger(MemoryManager memoryManager,
IOManager ioManager,
MutableObjectIterator<E> input,
AbstractInvokable parentTask,
TypeSerializerFactory<E> serializerFactory,
TypeComparator<E> comparator,
double memoryFraction,
int numSortBuffers,
int maxNumFileHandles,
float startSpillingFraction,
boolean noSpillingMemory,
boolean handleLargeRecords,
boolean objectReuseEnabled) |
UnilateralSortMerger(MemoryManager memoryManager,
List<MemorySegment> memory,
IOManager ioManager,
MutableObjectIterator<E> input,
AbstractInvokable parentTask,
TypeSerializerFactory<E> serializerFactory,
TypeComparator<E> comparator,
int numSortBuffers,
int maxNumFileHandles,
float startSpillingFraction,
boolean handleLargeRecords,
boolean objectReuseEnabled) |
UnilateralSortMerger(MemoryManager memoryManager,
List<MemorySegment> memory,
IOManager ioManager,
MutableObjectIterator<E> input,
AbstractInvokable parentTask,
TypeSerializerFactory<E> serializerFactory,
TypeComparator<E> comparator,
int numSortBuffers,
int maxNumFileHandles,
float startSpillingFraction,
boolean noSpillingMemory,
boolean handleLargeRecords,
boolean objectReuseEnabled) |
UnilateralSortMerger(MemoryManager memoryManager,
List<MemorySegment> memory,
IOManager ioManager,
MutableObjectIterator<E> input,
AbstractInvokable parentTask,
TypeSerializerFactory<E> serializerFactory,
TypeComparator<E> comparator,
int numSortBuffers,
int maxNumFileHandles,
float startSpillingFraction,
boolean noSpillingMemory,
boolean handleLargeRecords,
boolean objectReuseEnabled,
InMemorySorterFactory<E> inMemorySorterFactory) |
Modifier and Type | Method and Description |
---|---|
Class<? extends AbstractInvokable> |
StreamNode.getJobVertexClass() |
Modifier and Type | Method and Description |
---|---|
protected StreamNode |
StreamGraph.addNode(Integer vertexID,
String slotSharingGroup,
String coLocationGroup,
Class<? extends AbstractInvokable> vertexClass,
StreamOperatorFactory<?> operatorFactory,
String operatorName) |
Constructor and Description |
---|
StreamNode(Integer id,
String slotSharingGroup,
String coLocationGroup,
StreamOperator<?> operator,
String operatorName,
List<OutputSelector<?>> outputSelector,
Class<? extends AbstractInvokable> jobVertexClass) |
StreamNode(Integer id,
String slotSharingGroup,
String coLocationGroup,
StreamOperatorFactory<?> operatorFactory,
String operatorName,
List<OutputSelector<?>> outputSelector,
Class<? extends AbstractInvokable> jobVertexClass) |
Modifier and Type | Method and Description |
---|---|
static CheckpointedInputGate |
InputProcessorUtil.createCheckpointedInputGate(AbstractInvokable toNotifyOnCheckpoint,
StreamConfig config,
SubtaskCheckpointCoordinator checkpointCoordinator,
IndexedInputGate[] inputGates,
TaskIOMetricGroup taskIOMetricGroup,
String taskName) |
static CheckpointedInputGate[] |
InputProcessorUtil.createCheckpointedMultipleInputGate(AbstractInvokable toNotifyOnCheckpoint,
StreamConfig config,
SubtaskCheckpointCoordinator checkpointCoordinator,
TaskIOMetricGroup taskIOMetricGroup,
String taskName,
List<IndexedInputGate>... inputGates) |
Constructor and Description |
---|
CheckpointBarrierHandler(AbstractInvokable toNotifyOnCheckpoint) |
CheckpointBarrierTracker(int totalNumberOfInputChannels,
AbstractInvokable toNotifyOnCheckpoint) |
Modifier and Type | Class and Description |
---|---|
class |
AbstractTwoInputStreamTask<IN1,IN2,OUT>
Abstract class for executing a
TwoInputStreamOperator . |
class |
MultipleInputStreamTask<OUT>
A
StreamTask for executing a MultipleInputStreamOperator and supporting the
MultipleInputStreamOperator to select input for reading. |
class |
OneInputStreamTask<IN,OUT>
A
StreamTask for executing a OneInputStreamOperator . |
class |
SourceOperatorStreamTask<T>
A subclass of
StreamTask for executing the SourceOperator . |
class |
SourceStreamTask<OUT,SRC extends SourceFunction<OUT>,OP extends StreamSource<OUT,SRC>>
StreamTask for executing a StreamSource . |
class |
StreamIterationHead<OUT>
A special
StreamTask that is used for executing feedback edges. |
class |
StreamIterationTail<IN>
A special
StreamTask that is used for executing feedback edges. |
class |
StreamTask<OUT,OP extends StreamOperator<OUT>>
Base class for all streaming tasks.
|
class |
TwoInputStreamTask<IN1,IN2,OUT>
A
StreamTask for executing a TwoInputStreamOperator and supporting the TwoInputStreamOperator to select input for reading. |
Copyright © 2014–2021 The Apache Software Foundation. All rights reserved.