org.apache.flink.runtime.io.network.partition.hybrid

Class HybridShuffleConfiguration

java.lang.Object

org.apache.flink.runtime.io.network.partition.hybrid.HybridShuffleConfiguration

public class HybridShuffleConfiguration
extends Object

Configuration for hybrid shuffle mode.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HybridShuffleConfiguration.Builder Builder for HybridShuffleConfiguration.
static class	HybridShuffleConfiguration.SpillingStrategyType Type of HsSpillingStrategy.

Method Summary

Modifier and Type	Method and Description
static HybridShuffleConfiguration.Builder	builder(int numSubpartitions, int numBuffersPerRequest)
long	getBufferPoolSizeCheckIntervalMs() Check interval of buffer pool's size.
Duration	getBufferRequestTimeout() Maximum time to wait when requesting read buffers from the buffer pool before throwing an exception.
float	getFullStrategyNumBuffersTriggerSpillingRatio() When the number of unSpilled buffers equal to this ratio times pool size, trigger the spilling operation.
float	getFullStrategyReleaseBufferRatio() The proportion of buffers to be released.
float	getFullStrategyReleaseThreshold() When the number of buffers that have been requested exceeds this threshold, trigger the release operation.
int	getMaxBuffersReadAhead() Determine how many buffers to read ahead at most for each subpartition to prevent other consumers from starving.
int	getMaxRequestedBuffers()
long	getNumRetainedInMemoryRegionsMax() Max number of hybrid retained regions in memory.
int	getRegionGroupSizeInBytes() Segment size of hybrid spilled file data index.
float	getSelectiveStrategySpillBufferRatio() The proportion of buffers to be spilled.
float	getSelectiveStrategySpillThreshold() When the number of buffers that have been requested exceeds this threshold, trigger the spilling operation.
HybridShuffleConfiguration.SpillingStrategyType	getSpillingStrategyType() Get HybridShuffleConfiguration.SpillingStrategyType for hybrid shuffle mode.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

builder

public static HybridShuffleConfiguration.Builder builder(int numSubpartitions,
                                                         int numBuffersPerRequest)

getSpillingStrategyType

public HybridShuffleConfiguration.SpillingStrategyType getSpillingStrategyType()

Get HybridShuffleConfiguration.SpillingStrategyType for hybrid shuffle mode.

getMaxRequestedBuffers

public int getMaxRequestedBuffers()

getMaxBuffersReadAhead

public int getMaxBuffersReadAhead()

Determine how many buffers to read ahead at most for each subpartition to prevent other consumers from starving.

getBufferRequestTimeout

public Duration getBufferRequestTimeout()

Maximum time to wait when requesting read buffers from the buffer pool before throwing an exception.

getSelectiveStrategySpillThreshold

public float getSelectiveStrategySpillThreshold()

When the number of buffers that have been requested exceeds this threshold, trigger the spilling operation. Used by HsSelectiveSpillingStrategy.

getSelectiveStrategySpillBufferRatio

public float getSelectiveStrategySpillBufferRatio()

The proportion of buffers to be spilled. Used by HsSelectiveSpillingStrategy.

getFullStrategyNumBuffersTriggerSpillingRatio

public float getFullStrategyNumBuffersTriggerSpillingRatio()

When the number of unSpilled buffers equal to this ratio times pool size, trigger the spilling operation. Used by HsFullSpillingStrategy.

getFullStrategyReleaseThreshold

public float getFullStrategyReleaseThreshold()

When the number of buffers that have been requested exceeds this threshold, trigger the release operation. Used by HsFullSpillingStrategy.

getFullStrategyReleaseBufferRatio

public float getFullStrategyReleaseBufferRatio()

The proportion of buffers to be released. Used by HsFullSpillingStrategy.

getBufferPoolSizeCheckIntervalMs

public long getBufferPoolSizeCheckIntervalMs()

Check interval of buffer pool's size.

getRegionGroupSizeInBytes

public int getRegionGroupSizeInBytes()

Segment size of hybrid spilled file data index.

getNumRetainedInMemoryRegionsMax

public long getNumRetainedInMemoryRegionsMax()

Max number of hybrid retained regions in memory.