org.apache.flink.runtime.io.network.partition.hybrid

Interface HsSpillingInfoProvider

All Known Implementing Classes:

HsMemoryDataManager

public interface HsSpillingInfoProvider

This component is responsible for providing some information needed for the spill decision.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	HsSpillingInfoProvider.ConsumeStatus This enum represents buffer status of consume in hybrid shuffle mode.
static class	HsSpillingInfoProvider.ConsumeStatusWithId This class represents a pair of HsSpillingInfoProvider.ConsumeStatus and consumer id.
static class	HsSpillingInfoProvider.SpillStatus This enum represents buffer status of spill in hybrid shuffle mode.

Method Summary

Modifier and Type	Method and Description
Deque<BufferIndexAndChannel>	getBuffersInOrder(int subpartitionId, HsSpillingInfoProvider.SpillStatus spillStatus, HsSpillingInfoProvider.ConsumeStatusWithId consumeStatusWithId) Get all buffers with the expected status from the subpartition.
List<Integer>	getNextBufferIndexToConsume(HsConsumerId consumerId) Get all subpartition's next buffer index to consume of specific consumer.
int	getNumSubpartitions() Get the number of downstream consumers.
int	getNumTotalRequestedBuffers() Get total number of buffers requested from buffer pool.
int	getNumTotalUnSpillBuffers() Get total number of not decided to spill buffers.
int	getPoolSize() Get the current size of buffer pool.

Method Detail

getNumSubpartitions

int getNumSubpartitions()

Get the number of downstream consumers.

Returns:

Number of subpartitions.

getNextBufferIndexToConsume

List<Integer> getNextBufferIndexToConsume(HsConsumerId consumerId)

Get all subpartition's next buffer index to consume of specific consumer.

Parameters:

consumerId - of the target downstream consumer.

Returns:

A list containing all subpartition's next buffer index to consume of specific consumer, if the downstream subpartition view has not been registered, the corresponding return value is -1.

getBuffersInOrder

Deque<BufferIndexAndChannel> getBuffersInOrder(int subpartitionId,
                                               HsSpillingInfoProvider.SpillStatus spillStatus,
                                               HsSpillingInfoProvider.ConsumeStatusWithId consumeStatusWithId)

Get all buffers with the expected status from the subpartition.

Parameters:

subpartitionId - target buffers belong to.

spillStatus - expected buffer spill status.

consumeStatusWithId - expected buffer consume status and consumer id.

Returns:

all buffers satisfy specific status of this subpartition, This queue must be sorted according to bufferIndex from small to large, in other words, head is the buffer with the minimum bufferIndex in the current subpartition.

getNumTotalUnSpillBuffers

int getNumTotalUnSpillBuffers()

Get total number of not decided to spill buffers.

getNumTotalRequestedBuffers

int getNumTotalRequestedBuffers()

Get total number of buffers requested from buffer pool.

getPoolSize

int getPoolSize()

Get the current size of buffer pool.