Modifier and Type | Method and Description |
---|---|
ResultPartitionType |
InputGateDeploymentDescriptor.getConsumedPartitionType()
Returns the type of this input channel's consumed result partition.
|
ResultPartitionType |
ResultPartitionDeploymentDescriptor.getPartitionType() |
Constructor and Description |
---|
InputGateDeploymentDescriptor(IntermediateDataSetID consumedResultId,
ResultPartitionType consumedPartitionType,
IndexRange consumedSubpartitionIndexRange,
int numberOfInputChannels,
List<TaskDeploymentDescriptor.MaybeOffloaded<TaskDeploymentDescriptorFactory.ShuffleDescriptorGroup>> serializedInputChannels) |
InputGateDeploymentDescriptor(IntermediateDataSetID consumedResultId,
ResultPartitionType consumedPartitionType,
int consumedSubpartitionIndex,
TaskDeploymentDescriptorFactory.ShuffleDescriptorAndIndex[] inputChannels) |
Modifier and Type | Method and Description |
---|---|
ResultPartitionType |
IntermediateResult.getResultType() |
ResultPartitionType |
IntermediateResultPartition.getResultType() |
Modifier and Type | Method and Description |
---|---|
boolean |
MarkPartitionFinishedStrategy.needMarkPartitionFinished(ResultPartitionType resultPartitionType)
Decide whether partition needs to be marked as finished.
|
Constructor and Description |
---|
IntermediateResult(IntermediateDataSet intermediateDataSet,
ExecutionJobVertex producer,
int numParallelProducers,
ResultPartitionType resultType) |
Modifier and Type | Field and Description |
---|---|
protected ResultPartitionType |
ResultPartition.partitionType
Type of this partition.
|
Modifier and Type | Method and Description |
---|---|
ResultPartitionType |
ResultPartition.getPartitionType()
Returns the type of this result partition.
|
static ResultPartitionType |
ResultPartitionType.valueOf(String name)
Returns the enum constant of this type with the specified name.
|
static ResultPartitionType[] |
ResultPartitionType.values()
Returns an array containing the constants of this enum type, in
the order they are declared.
|
Modifier and Type | Method and Description |
---|---|
ResultPartition |
ResultPartitionFactory.create(String taskNameWithSubtaskAndId,
int partitionIndex,
ResultPartitionID id,
ResultPartitionType type,
int numberOfPartitions,
int numberOfSubpartitions,
int maxParallelism,
boolean isBroadcast,
ShuffleDescriptor shuffleDescriptor,
SupplierWithException<BufferPool,IOException> bufferPoolFactory,
boolean isNumberOfPartitionConsumerUndefined) |
Constructor and Description |
---|
BoundedBlockingResultPartition(String owningTaskName,
int partitionIndex,
ResultPartitionID partitionId,
ResultPartitionType partitionType,
ResultSubpartition[] subpartitions,
int numTargetKeyGroups,
ResultPartitionManager partitionManager,
BufferCompressor bufferCompressor,
SupplierWithException<BufferPool,IOException> bufferPoolFactory) |
BufferWritingResultPartition(String owningTaskName,
int partitionIndex,
ResultPartitionID partitionId,
ResultPartitionType partitionType,
ResultSubpartition[] subpartitions,
int numTargetKeyGroups,
ResultPartitionManager partitionManager,
BufferCompressor bufferCompressor,
SupplierWithException<BufferPool,IOException> bufferPoolFactory) |
PipelinedResultPartition(String owningTaskName,
int partitionIndex,
ResultPartitionID partitionId,
ResultPartitionType partitionType,
ResultSubpartition[] subpartitions,
int numTargetKeyGroups,
ResultPartitionManager partitionManager,
BufferCompressor bufferCompressor,
SupplierWithException<BufferPool,IOException> bufferPoolFactory) |
ResultPartition(String owningTaskName,
int partitionIndex,
ResultPartitionID partitionId,
ResultPartitionType partitionType,
int numSubpartitions,
int numTargetKeyGroups,
ResultPartitionManager partitionManager,
BufferCompressor bufferCompressor,
SupplierWithException<BufferPool,IOException> bufferPoolFactory) |
SortMergeResultPartition(String owningTaskName,
int partitionIndex,
ResultPartitionID partitionId,
ResultPartitionType partitionType,
int numSubpartitions,
int numTargetKeyGroups,
BatchShuffleReadBufferPool readBufferPool,
Executor readIOExecutor,
ResultPartitionManager partitionManager,
String resultFileBasePath,
BufferCompressor bufferCompressor,
SupplierWithException<BufferPool,IOException> bufferPoolFactory) |
Modifier and Type | Method and Description |
---|---|
ResultPartitionType |
SingleInputGate.getConsumedPartitionType()
Returns the type of this input channel's consumed result partition.
|
Modifier and Type | Method and Description |
---|---|
static GateBuffersSpec |
InputGateSpecUtils.createGateBuffersSpec(Optional<Integer> configuredMaxRequiredBuffersPerGate,
int configuredNetworkBuffersPerChannel,
int configuredFloatingNetworkBuffersPerGate,
ResultPartitionType partitionType,
int numInputChannels,
boolean enableTieredStorage) |
Constructor and Description |
---|
SingleInputGate(String owningTaskName,
int gateIndex,
IntermediateDataSetID consumedResultId,
ResultPartitionType consumedPartitionType,
int numberOfInputChannels,
PartitionProducerStateProvider partitionProducerStateProvider,
SupplierWithException<BufferPool,IOException> bufferPoolFactory,
BufferDecompressor bufferDecompressor,
MemorySegmentProvider memorySegmentProvider,
int segmentSize,
ThroughputCalculator throughputCalculator,
BufferDebloater bufferDebloater) |
Constructor and Description |
---|
HsResultPartition(String owningTaskName,
int partitionIndex,
ResultPartitionID partitionId,
ResultPartitionType partitionType,
int numSubpartitions,
int numTargetKeyGroups,
BatchShuffleReadBufferPool readBufferPool,
ScheduledExecutorService readIOExecutor,
ResultPartitionManager partitionManager,
String dataFileBashPath,
int networkBufferSize,
HybridShuffleConfiguration hybridShuffleConfiguration,
BufferCompressor bufferCompressor,
boolean isBroadcastOnly,
SupplierWithException<BufferPool,IOException> bufferPoolFactory) |
Modifier and Type | Method and Description |
---|---|
TieredResultPartition |
TieredResultPartitionFactory.createTieredResultPartition(String owningTaskName,
int partitionIndex,
ResultPartitionID partitionId,
ResultPartitionType partitionType,
int numPartitions,
int numSubpartitions,
int maxParallelism,
int bufferSizeBytes,
Boolean isBroadCastOnly,
ResultPartitionManager partitionManager,
BufferCompressor bufferCompressor,
List<TierShuffleDescriptor> tierShuffleDescriptors,
SupplierWithException<BufferPool,IOException> bufferPoolFactory,
FileChannelManager fileChannelManager,
BatchShuffleReadBufferPool batchShuffleReadBufferPool,
ScheduledExecutorService batchShuffleReadIOExecutor,
boolean isNumberOfPartitionConsumerUndefined) |
Constructor and Description |
---|
TieredResultPartition(String owningTaskName,
int partitionIndex,
ResultPartitionID partitionId,
ResultPartitionType partitionType,
int numSubpartitions,
int numTargetKeyGroups,
ResultPartitionManager partitionManager,
BufferCompressor bufferCompressor,
SupplierWithException<BufferPool,IOException> bufferPoolFactory,
TieredStorageProducerClient tieredStorageProducerClient,
TieredStorageResourceRegistry tieredStorageResourceRegistry,
TieredStorageNettyServiceImpl nettyService,
List<TieredStorageMemorySpec> tieredStorageMemorySpecs,
TieredStorageMemoryManager storageMemoryManager) |
Modifier and Type | Method and Description |
---|---|
ResultPartitionType |
IntermediateDataSet.getResultType() |
Modifier and Type | Method and Description |
---|---|
JobEdge |
JobVertex.connectNewDataSetAsInput(JobVertex input,
DistributionPattern distPattern,
ResultPartitionType partitionType) |
JobEdge |
JobVertex.connectNewDataSetAsInput(JobVertex input,
DistributionPattern distPattern,
ResultPartitionType partitionType,
boolean isBroadcast) |
JobEdge |
JobVertex.connectNewDataSetAsInput(JobVertex input,
DistributionPattern distPattern,
ResultPartitionType partitionType,
IntermediateDataSetID intermediateDataSetId,
boolean isBroadcast) |
IntermediateDataSet |
JobVertex.getOrCreateResultDataSet(IntermediateDataSetID id,
ResultPartitionType partitionType) |
Constructor and Description |
---|
IntermediateDataSet(IntermediateDataSetID id,
ResultPartitionType resultType,
JobVertex producer) |
Modifier and Type | Method and Description |
---|---|
ResultPartitionType |
DefaultLogicalResult.getResultType() |
Modifier and Type | Method and Description |
---|---|
ResultPartitionType |
ConsumedPartitionGroup.getResultPartitionType() |
ResultPartitionType |
ConsumerVertexGroup.getResultPartitionType() |
Modifier and Type | Method and Description |
---|---|
static ConsumedPartitionGroup |
ConsumedPartitionGroup.fromMultiplePartitions(int numConsumers,
List<IntermediateResultPartitionID> resultPartitions,
ResultPartitionType resultPartitionType) |
static ConsumerVertexGroup |
ConsumerVertexGroup.fromMultipleVertices(List<ExecutionVertexID> vertices,
ResultPartitionType resultPartitionType) |
static ConsumedPartitionGroup |
ConsumedPartitionGroup.fromSinglePartition(int numConsumers,
IntermediateResultPartitionID resultPartition,
ResultPartitionType resultPartitionType) |
static ConsumerVertexGroup |
ConsumerVertexGroup.fromSingleVertex(ExecutionVertexID vertex,
ResultPartitionType resultPartitionType) |
Modifier and Type | Method and Description |
---|---|
ResultPartitionType |
PartitionDescriptor.getPartitionType() |
Modifier and Type | Method and Description |
---|---|
Map<IntermediateDataSetID,ResultPartitionType> |
TaskInputsOutputsDescriptor.getInputPartitionTypes() |
Map<IntermediateDataSetID,ResultPartitionType> |
TaskInputsOutputsDescriptor.getPartitionTypes() |
Modifier and Type | Method and Description |
---|---|
static org.apache.commons.lang3.tuple.Pair<Integer,Integer> |
NettyShuffleUtils.getMinMaxNetworkBuffersPerResultPartition(int configuredNetworkBuffersPerChannel,
int numFloatingBuffersPerGate,
int sortShuffleMinParallelism,
int sortShuffleMinBuffers,
int numSubpartitions,
boolean enableTieredStorage,
int tieredStoreExclusiveBuffers,
ResultPartitionType type)
Calculates and returns local network buffer pool size used by the result partition.
|
Modifier and Type | Method and Description |
---|---|
static int |
NettyShuffleUtils.computeNetworkBuffersForAnnouncing(int numBuffersPerChannel,
int numFloatingBuffersPerGate,
Optional<Integer> maxRequiredBuffersPerGate,
int sortShuffleMinParallelism,
int sortShuffleMinBuffers,
Map<IntermediateDataSetID,Integer> inputChannelNums,
Map<IntermediateDataSetID,Integer> partitionReuseCount,
Map<IntermediateDataSetID,Integer> subpartitionNums,
Map<IntermediateDataSetID,ResultPartitionType> inputPartitionTypes,
Map<IntermediateDataSetID,ResultPartitionType> partitionTypes) |
static int |
NettyShuffleUtils.computeNetworkBuffersForAnnouncing(int numBuffersPerChannel,
int numFloatingBuffersPerGate,
Optional<Integer> maxRequiredBuffersPerGate,
int sortShuffleMinParallelism,
int sortShuffleMinBuffers,
Map<IntermediateDataSetID,Integer> inputChannelNums,
Map<IntermediateDataSetID,Integer> partitionReuseCount,
Map<IntermediateDataSetID,Integer> subpartitionNums,
Map<IntermediateDataSetID,ResultPartitionType> inputPartitionTypes,
Map<IntermediateDataSetID,ResultPartitionType> partitionTypes) |
static TaskInputsOutputsDescriptor |
TaskInputsOutputsDescriptor.from(int inputGateNums,
Map<IntermediateDataSetID,Integer> inputChannelNums,
Map<IntermediateDataSetID,Integer> partitionReuseCount,
Map<IntermediateDataSetID,Integer> subpartitionNums,
Map<IntermediateDataSetID,ResultPartitionType> inputPartitionTypes,
Map<IntermediateDataSetID,ResultPartitionType> partitionTypes) |
static TaskInputsOutputsDescriptor |
TaskInputsOutputsDescriptor.from(int inputGateNums,
Map<IntermediateDataSetID,Integer> inputChannelNums,
Map<IntermediateDataSetID,Integer> partitionReuseCount,
Map<IntermediateDataSetID,Integer> subpartitionNums,
Map<IntermediateDataSetID,ResultPartitionType> inputPartitionTypes,
Map<IntermediateDataSetID,ResultPartitionType> partitionTypes) |
Constructor and Description |
---|
PartitionDescriptor(IntermediateDataSetID resultId,
int totalNumberOfPartitions,
IntermediateResultPartitionID partitionId,
ResultPartitionType partitionType,
int numberOfSubpartitions,
int connectionIndex,
boolean isBroadcast,
boolean isAllToAllDistribution,
boolean isNumberOfPartitionConsumerUndefined) |
Modifier and Type | Method and Description |
---|---|
ResultPartitionType |
Result.getResultType() |
Modifier and Type | Method and Description |
---|---|
ResultPartitionType |
NonChainedOutput.getPartitionType() |
Modifier and Type | Method and Description |
---|---|
void |
NonChainedOutput.setPartitionType(ResultPartitionType partitionType) |
Constructor and Description |
---|
NonChainedOutput(boolean supportsUnalignedCheckpoints,
int sourceNodeId,
int consumerParallelism,
int consumerMaxParallelism,
long bufferTimeout,
boolean isPersistentDataSet,
IntermediateDataSetID dataSetId,
OutputTag<?> outputTag,
StreamPartitioner<?> partitioner,
ResultPartitionType partitionType) |
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.