org.apache.flink.runtime.io.network.partition.hybrid.tiered.common

Class TieredStorageConfiguration

java.lang.Object

org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageConfiguration

public class TieredStorageConfiguration
extends Object

Configurations for the Tiered Storage.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TieredStorageConfiguration.Builder The builder for TieredStorageConfiguration.

Constructor Summary

Constructors

Constructor and Description

TieredStorageConfiguration(String remoteStorageBasePath, int tieredStorageBufferSize, int minBuffersPerGate, int minBuffersPerResultPartition, int memoryTierExclusiveBuffers, int diskTierExclusiveBuffers, int remoteTierExclusiveBuffers, int accumulatorExclusiveBuffers, int memoryTierNumBytesPerSegment, int diskTierNumBytesPerSegment, int remoteTierNumBytesPerSegment, float numBuffersTriggerFlushRatio, java.time.Duration diskIOSchedulerRequestTimeout, float minReserveDiskSpaceFraction, boolean memoryDecouplingEnabled, long poolSizeCheckInterval, List<TierFactory> tierFactories, List<Integer> tierExclusiveBuffers)

Method Summary

Modifier and Type	Method and Description
static TieredStorageConfiguration.Builder	builder(int tieredStorageBufferSize, String remoteStorageBasePath)
static TieredStorageConfiguration.Builder	builder(String remoteStorageBasePath)
int	getAccumulatorExclusiveBuffers() Get exclusive buffer number of accumulator.
java.time.Duration	getDiskIOSchedulerBufferRequestTimeout() Maximum time to wait when requesting read buffers from the buffer pool before throwing an exception in DiskIOScheduler.
int	getDiskTierExclusiveBuffers() Get exclusive buffer number of disk tier.
int	getDiskTierNumBytesPerSegment() Get the segment size of disk tier.
List<Integer>	getEachTierExclusiveBufferNum() Get exclusive buffer number of each tier.
boolean	getMemoryDecouplingEnabled() Whether the memory is decoupled from job topology.
int	getMemoryTierExclusiveBuffers() Get exclusive buffer number of memory tier.
int	getMemoryTierNumBytesPerSegment() Get the segment size of memory tier.
int	getMinBuffersPerGate() Get the number of minimum buffers per gate.
int	getMinBuffersPerResultPartition() *
float	getMinReserveDiskSpaceFraction() Minimum reserved disk space fraction in disk tier.
float	getNumBuffersTriggerFlushRatio() When the number of buffers that have been requested exceeds this threshold, trigger the flushing operation in each TierProducerAgent.
long	getPoolSizeCheckInterval() Get the interval in which the buffer pool size is checked.
String	getRemoteStorageBasePath() Get the base path on remote storage.
int	getRemoteTierExclusiveBuffers() Get exclusive buffer number of remote tier.
int	getRemoteTierNumBytesPerSegment() Get the segment size of remote tier.
int	getTieredStorageBufferSize() Get the buffer size in tiered storage.
List<TierFactory>	getTierFactories()
int	getTotalExclusiveBufferNum() Get the total exclusive buffer number.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TieredStorageConfiguration

public TieredStorageConfiguration(String remoteStorageBasePath,
                                  int tieredStorageBufferSize,
                                  int minBuffersPerGate,
                                  int minBuffersPerResultPartition,
                                  int memoryTierExclusiveBuffers,
                                  int diskTierExclusiveBuffers,
                                  int remoteTierExclusiveBuffers,
                                  int accumulatorExclusiveBuffers,
                                  int memoryTierNumBytesPerSegment,
                                  int diskTierNumBytesPerSegment,
                                  int remoteTierNumBytesPerSegment,
                                  float numBuffersTriggerFlushRatio,
                                  java.time.Duration diskIOSchedulerRequestTimeout,
                                  float minReserveDiskSpaceFraction,
                                  boolean memoryDecouplingEnabled,
                                  long poolSizeCheckInterval,
                                  List<TierFactory> tierFactories,
                                  List<Integer> tierExclusiveBuffers)

Method Detail

builder

public static TieredStorageConfiguration.Builder builder(String remoteStorageBasePath)

builder

public static TieredStorageConfiguration.Builder builder(int tieredStorageBufferSize,
                                                         String remoteStorageBasePath)

getRemoteStorageBasePath

public String getRemoteStorageBasePath()

Get the base path on remote storage.

Returns:

string if the remote storage path is configured otherwise null.

getTieredStorageBufferSize

public int getTieredStorageBufferSize()

Get the buffer size in tiered storage.

Returns:

the buffer size.

getMinBuffersPerGate

public int getMinBuffersPerGate()

Get the number of minimum buffers per gate.

Returns:

the buffer number.

getMinBuffersPerResultPartition

public int getMinBuffersPerResultPartition()

*

Get the number of minimum buffers per result partition.

Returns:

the buffer number.

getMemoryTierExclusiveBuffers

public int getMemoryTierExclusiveBuffers()

Get exclusive buffer number of memory tier.

Returns:

the buffer number.

getDiskTierExclusiveBuffers

public int getDiskTierExclusiveBuffers()

Get exclusive buffer number of disk tier.

Returns:

the buffer number.

getRemoteTierExclusiveBuffers

public int getRemoteTierExclusiveBuffers()

Get exclusive buffer number of remote tier.

Returns:

the buffer number.

getAccumulatorExclusiveBuffers

public int getAccumulatorExclusiveBuffers()

Get exclusive buffer number of accumulator.

The buffer number is used to compare with the subpartition number to determine the type of BufferAccumulator.

If the exclusive buffer number is larger than (subpartitionNum + 1), the accumulator will use HashBufferAccumulator. If the exclusive buffer number is equal to or smaller than (subpartitionNum + 1), the accumulator will use SortBufferAccumulator

Returns:

the buffer number.

getMemoryTierNumBytesPerSegment

public int getMemoryTierNumBytesPerSegment()

Get the segment size of memory tier.

Returns:

segment size.

getDiskTierNumBytesPerSegment

public int getDiskTierNumBytesPerSegment()

Get the segment size of disk tier.

Returns:

segment size.

getRemoteTierNumBytesPerSegment

public int getRemoteTierNumBytesPerSegment()

Get the segment size of remote tier.

Returns:

segment size.

getNumBuffersTriggerFlushRatio

public float getNumBuffersTriggerFlushRatio()

When the number of buffers that have been requested exceeds this threshold, trigger the flushing operation in each TierProducerAgent.

Returns:

flush ratio.

getDiskIOSchedulerBufferRequestTimeout

public java.time.Duration getDiskIOSchedulerBufferRequestTimeout()

Maximum time to wait when requesting read buffers from the buffer pool before throwing an exception in DiskIOScheduler.

Returns:

timeout duration.

getMinReserveDiskSpaceFraction

public float getMinReserveDiskSpaceFraction()

Minimum reserved disk space fraction in disk tier.

Returns:

the fraction.

getMemoryDecouplingEnabled

public boolean getMemoryDecouplingEnabled()

Whether the memory is decoupled from job topology.

Returns:

true if enabled, false otherwise.

getPoolSizeCheckInterval

public long getPoolSizeCheckInterval()

Get the interval in which the buffer pool size is checked.

Returns:

the interval.

getTotalExclusiveBufferNum

public int getTotalExclusiveBufferNum()

Get the total exclusive buffer number.

Returns:

the total exclusive buffer number.

getEachTierExclusiveBufferNum

public List<Integer> getEachTierExclusiveBufferNum()

Get exclusive buffer number of each tier.

Returns:

buffer number of each tier.

getTierFactories

public List<TierFactory> getTierFactories()