Package | Description |
---|---|
org.apache.flink.streaming.api.datastream | |
org.apache.flink.streaming.api.graph | |
org.apache.flink.streaming.api.transformations | |
org.apache.flink.streaming.runtime.io | |
org.apache.flink.streaming.runtime.io.recovery | |
org.apache.flink.streaming.runtime.partitioner | |
org.apache.flink.streaming.runtime.tasks |
This package contains classes that realize streaming tasks.
|
org.apache.flink.table.runtime.partitioner |
Modifier and Type | Method and Description |
---|---|
protected DataStream<T> |
DataStream.setConnectionType(StreamPartitioner<T> partitioner)
Internal function for setting the partitioner for the DataStream.
|
protected DataStream<T> |
KeyedStream.setConnectionType(StreamPartitioner<T> partitioner) |
Modifier and Type | Method and Description |
---|---|
StreamPartitioner<?> |
StreamEdge.getPartitioner() |
StreamPartitioner<?> |
NonChainedOutput.getPartitioner() |
Modifier and Type | Method and Description |
---|---|
void |
StreamGraph.addVirtualPartitionNode(Integer originalId,
Integer virtualId,
StreamPartitioner<?> partitioner,
StreamExchangeMode exchangeMode)
Adds a new virtual node that is used to connect a downstream vertex to an input with a
certain partitioning.
|
void |
StreamEdge.setPartitioner(StreamPartitioner<?> partitioner) |
void |
NonChainedOutput.setPartitioner(StreamPartitioner<?> partitioner) |
Constructor and Description |
---|
NonChainedOutput(boolean supportsUnalignedCheckpoints,
int sourceNodeId,
int consumerParallelism,
int consumerMaxParallelism,
long bufferTimeout,
boolean isPersistentDataSet,
IntermediateDataSetID dataSetId,
OutputTag<?> outputTag,
StreamPartitioner<?> partitioner,
ResultPartitionType partitionType) |
StreamEdge(StreamNode sourceVertex,
StreamNode targetVertex,
int typeNumber,
long bufferTimeout,
StreamPartitioner<?> outputPartitioner,
OutputTag outputTag,
StreamExchangeMode exchangeMode,
int uniqueId,
IntermediateDataSetID intermediateDatasetId) |
StreamEdge(StreamNode sourceVertex,
StreamNode targetVertex,
int typeNumber,
StreamPartitioner<?> outputPartitioner,
OutputTag outputTag) |
StreamEdge(StreamNode sourceVertex,
StreamNode targetVertex,
int typeNumber,
StreamPartitioner<?> outputPartitioner,
OutputTag outputTag,
StreamExchangeMode exchangeMode,
int uniqueId,
IntermediateDataSetID intermediateDatasetId) |
Modifier and Type | Method and Description |
---|---|
StreamPartitioner<T> |
PartitionTransformation.getPartitioner()
Returns the
StreamPartitioner that must be used for partitioning the elements of the
input Transformation . |
Constructor and Description |
---|
PartitionTransformation(Transformation<T> input,
StreamPartitioner<T> partitioner)
Creates a new
PartitionTransformation from the given input and StreamPartitioner . |
PartitionTransformation(Transformation<T> input,
StreamPartitioner<T> partitioner,
StreamExchangeMode exchangeMode)
Creates a new
PartitionTransformation from the given input and StreamPartitioner . |
Modifier and Type | Method and Description |
---|---|
static <T> StreamTaskInput<T> |
StreamTaskNetworkInputFactory.create(CheckpointedInputGate checkpointedInputGate,
TypeSerializer<T> inputSerializer,
IOManager ioManager,
StatusWatermarkValve statusWatermarkValve,
int inputIndex,
InflightDataRescalingDescriptor rescalingDescriptorinflightDataRescalingDescriptor,
Function<Integer,StreamPartitioner<?>> gatePartitioners,
TaskInfo taskInfo,
StreamTask.CanEmitBatchOfRecordsChecker canEmitBatchOfRecords)
Factory method for
StreamTaskNetworkInput or RescalingStreamTaskNetworkInput
depending on InflightDataRescalingDescriptor . |
static <IN1,IN2> StreamMultipleInputProcessor |
StreamTwoInputProcessorFactory.create(TaskInvokable ownerTask,
CheckpointedInputGate[] checkpointedInputGates,
IOManager ioManager,
MemoryManager memoryManager,
TaskIOMetricGroup taskIOMetricGroup,
TwoInputStreamOperator<IN1,IN2,?> streamOperator,
WatermarkGauge input1WatermarkGauge,
WatermarkGauge input2WatermarkGauge,
OperatorChain<?,?> operatorChain,
StreamConfig streamConfig,
Configuration taskManagerConfig,
Configuration jobConfig,
ExecutionConfig executionConfig,
ClassLoader userClassloader,
Counter numRecordsIn,
InflightDataRescalingDescriptor inflightDataRescalingDescriptor,
Function<Integer,StreamPartitioner<?>> gatePartitioners,
TaskInfo taskInfo,
StreamTask.CanEmitBatchOfRecordsChecker canEmitBatchOfRecords) |
static StreamMultipleInputProcessor |
StreamMultipleInputProcessorFactory.create(TaskInvokable ownerTask,
CheckpointedInputGate[] checkpointedInputGates,
StreamConfig.InputConfig[] configuredInputs,
IOManager ioManager,
MemoryManager memoryManager,
TaskIOMetricGroup ioMetricGroup,
Counter mainOperatorRecordsIn,
MultipleInputStreamOperator<?> mainOperator,
WatermarkGauge[] inputWatermarkGauges,
StreamConfig streamConfig,
Configuration taskManagerConfig,
Configuration jobConfig,
ExecutionConfig executionConfig,
ClassLoader userClassloader,
OperatorChain<?,?> operatorChain,
InflightDataRescalingDescriptor inflightDataRescalingDescriptor,
Function<Integer,StreamPartitioner<?>> gatePartitioners,
TaskInfo taskInfo,
StreamTask.CanEmitBatchOfRecordsChecker canEmitBatchOfRecords) |
Constructor and Description |
---|
RescalingStreamTaskNetworkInput(CheckpointedInputGate checkpointedInputGate,
TypeSerializer<T> inputSerializer,
IOManager ioManager,
StatusWatermarkValve statusWatermarkValve,
int inputIndex,
InflightDataRescalingDescriptor inflightDataRescalingDescriptor,
Function<Integer,StreamPartitioner<?>> gatePartitioners,
TaskInfo taskInfo,
StreamTask.CanEmitBatchOfRecordsChecker canEmitBatchOfRecords) |
Modifier and Type | Class and Description |
---|---|
class |
BroadcastPartitioner<T>
Partitioner that selects all the output channels.
|
class |
CustomPartitionerWrapper<K,T>
Partitioner that selects the channel with a user defined partitioner function on a key.
|
class |
ForwardForConsecutiveHashPartitioner<T>
If there are multiple consecutive and the same hash shuffles, SQL planner will change them except
the first one to use forward partitioner, so that these operators can be chained to reduce
unnecessary shuffles.
|
class |
ForwardForUnspecifiedPartitioner<T>
When the parallelism of both upstream and downstream is
ExecutionConfig.PARALLELISM_DEFAULT and the edge's partitioner is not specified
(partitioner==null), the edge's partitioner will be set to FORWARD by default(See StreamGraph.createActualEdge(java.lang.Integer, java.lang.Integer, int, org.apache.flink.streaming.runtime.partitioner.StreamPartitioner<?>, org.apache.flink.util.OutputTag, org.apache.flink.streaming.api.transformations.StreamExchangeMode, org.apache.flink.runtime.jobgraph.IntermediateDataSetID) method for details). |
class |
ForwardPartitioner<T>
Partitioner that forwards elements only to the locally running downstream operation.
|
class |
GlobalPartitioner<T>
Partitioner that sends all elements to the downstream operator with subtask ID=0.
|
class |
KeyGroupStreamPartitioner<T,K>
Partitioner selects the target channel based on the key group index.
|
class |
RebalancePartitioner<T>
Partitioner that distributes the data equally by cycling through the output channels.
|
class |
RescalePartitioner<T>
Partitioner that distributes the data equally by cycling through the output channels.
|
class |
ShufflePartitioner<T>
Partitioner that distributes the data equally by selecting one output channel randomly.
|
Modifier and Type | Method and Description |
---|---|
StreamPartitioner<T> |
RebalancePartitioner.copy() |
StreamPartitioner<T> |
ForwardForConsecutiveHashPartitioner.copy() |
StreamPartitioner<T> |
KeyGroupStreamPartitioner.copy() |
StreamPartitioner<T> |
RescalePartitioner.copy() |
StreamPartitioner<T> |
GlobalPartitioner.copy() |
StreamPartitioner<T> |
ShufflePartitioner.copy() |
StreamPartitioner<T> |
BroadcastPartitioner.copy() |
StreamPartitioner<T> |
CustomPartitionerWrapper.copy() |
StreamPartitioner<T> |
ForwardPartitioner.copy() |
abstract StreamPartitioner<T> |
StreamPartitioner.copy() |
StreamPartitioner<T> |
ForwardForUnspecifiedPartitioner.copy() |
StreamPartitioner<T> |
ForwardForConsecutiveHashPartitioner.getHashPartitioner() |
Constructor and Description |
---|
ForwardForConsecutiveHashPartitioner(StreamPartitioner<T> hashPartitioner)
Create a new ForwardForConsecutiveHashPartitioner.
|
Modifier and Type | Method and Description |
---|---|
protected void |
MultipleInputStreamTask.createInputProcessor(List<IndexedInputGate>[] inputGates,
StreamConfig.InputConfig[] inputs,
WatermarkGauge[] inputWatermarkGauges,
Function<Integer,StreamPartitioner<?>> gatePartitioners) |
protected void |
TwoInputStreamTask.createInputProcessor(List<IndexedInputGate> inputGates1,
List<IndexedInputGate> inputGates2,
Function<Integer,StreamPartitioner<?>> gatePartitioners) |
protected abstract void |
AbstractTwoInputStreamTask.createInputProcessor(List<IndexedInputGate> inputGates1,
List<IndexedInputGate> inputGates2,
Function<Integer,StreamPartitioner<?>> gatePartitioners) |
Modifier and Type | Class and Description |
---|---|
class |
BinaryHashPartitioner
Hash partitioner for
BinaryRowData . |
Modifier and Type | Method and Description |
---|---|
StreamPartitioner<RowData> |
BinaryHashPartitioner.copy() |
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.