org.apache.flink.runtime.taskexecutor

Class TaskManagerServices

java.lang.Object

org.apache.flink.runtime.taskexecutor.TaskManagerServices

public class TaskManagerServices
extends Object

Container for TaskExecutor services such as the MemoryManager, IOManager, NetworkEnvironment. All services are exclusive to a single TaskExecutor. Consequently, the respective TaskExecutor is responsible for closing them.

Field Summary

Fields

Modifier and Type	Field and Description
static String	LOCAL_STATE_SUB_DIRECTORY_ROOT

Method Summary

Modifier and Type	Method and Description
static long	calculateHeapSizeMB(long totalJavaMemorySizeMB, Configuration config) Calculates the amount of heap memory to use (to set via -Xmx and -Xms) based on the total memory to use and the given configuration parameters.
static long	calculateNetworkBufferMemory(long totalJavaMemorySize, Configuration config) Calculates the amount of memory used for network buffers based on the total memory to use and the according configuration parameters.
static long	calculateNetworkBufferMemory(TaskManagerServicesConfiguration tmConfig, long maxJvmHeapMemory) Calculates the amount of memory used for network buffers inside the current JVM instance based on the available heap or the max heap size and the according configuration parameters.
static TaskManagerServices	fromConfiguration(TaskManagerServicesConfiguration taskManagerServicesConfiguration, ResourceID resourceID, Executor taskIOExecutor, long freeHeapMemoryWithDefrag, long maxJvmHeapMemory) Creates and returns the task manager services.
BroadcastVariableManager	getBroadcastVariableManager()
FileCache	getFileCache()
IOManager	getIOManager()
JobLeaderService	getJobLeaderService()
JobManagerTable	getJobManagerTable()
MemoryManager	getMemoryManager()
NetworkEnvironment	getNetworkEnvironment()
TaskManagerLocation	getTaskManagerLocation()
TaskExecutorLocalStateStoresManager	getTaskManagerStateStore()
TaskSlotTable	getTaskSlotTable()
void	shutDown() Shuts the TaskExecutor services down.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOCAL_STATE_SUB_DIRECTORY_ROOT

@VisibleForTesting
public static final String LOCAL_STATE_SUB_DIRECTORY_ROOT

See Also:

Constant Field Values

Method Detail

getMemoryManager

public MemoryManager getMemoryManager()

getIOManager

public IOManager getIOManager()

getNetworkEnvironment

public NetworkEnvironment getNetworkEnvironment()

getTaskManagerLocation

public TaskManagerLocation getTaskManagerLocation()

getBroadcastVariableManager

public BroadcastVariableManager getBroadcastVariableManager()

getFileCache

public FileCache getFileCache()

getTaskSlotTable

public TaskSlotTable getTaskSlotTable()

getJobManagerTable

public JobManagerTable getJobManagerTable()

getJobLeaderService

public JobLeaderService getJobLeaderService()

getTaskManagerStateStore

public TaskExecutorLocalStateStoresManager getTaskManagerStateStore()

shutDown

public void shutDown()
              throws FlinkException

Shuts the TaskExecutor services down.

Throws:

FlinkException

fromConfiguration

public static TaskManagerServices fromConfiguration(TaskManagerServicesConfiguration taskManagerServicesConfiguration,
                                                    ResourceID resourceID,
                                                    Executor taskIOExecutor,
                                                    long freeHeapMemoryWithDefrag,
                                                    long maxJvmHeapMemory)
                                             throws Exception

Creates and returns the task manager services.

Parameters:

resourceID - resource ID of the task manager

taskManagerServicesConfiguration - task manager configuration

taskIOExecutor - executor for async IO operations.

freeHeapMemoryWithDefrag - an estimate of the size of the free heap memory

maxJvmHeapMemory - the maximum JVM heap size

Returns:

task manager components

Throws:

Exception

calculateNetworkBufferMemory

public static long calculateNetworkBufferMemory(long totalJavaMemorySize,
                                                Configuration config)

Calculates the amount of memory used for network buffers based on the total memory to use and the according configuration parameters.

The following configuration parameters are involved:

• TaskManagerOptions.NETWORK_BUFFERS_MEMORY_FRACTION,
• TaskManagerOptions.NETWORK_BUFFERS_MEMORY_MIN,
• TaskManagerOptions.NETWORK_BUFFERS_MEMORY_MAX, and
• TaskManagerOptions.NETWORK_NUM_BUFFERS (fallback if the ones above do not exist)

.

Parameters:

totalJavaMemorySize - overall available memory to use (heap and off-heap, in bytes)

config - configuration object

Returns:

memory to use for network buffers (in bytes); at least one memory segment

calculateNetworkBufferMemory

public static long calculateNetworkBufferMemory(TaskManagerServicesConfiguration tmConfig,
                                                long maxJvmHeapMemory)

Calculates the amount of memory used for network buffers inside the current JVM instance based on the available heap or the max heap size and the according configuration parameters.

For containers or when started via scripts, if started with a memory limit and set to use off-heap memory, the maximum heap size for the JVM is adjusted accordingly and we are able to extract the intended values from this.

The following configuration parameters are involved:

• TaskManagerOptions.MANAGED_MEMORY_FRACTION,
• TaskManagerOptions.NETWORK_BUFFERS_MEMORY_FRACTION,
• TaskManagerOptions.NETWORK_BUFFERS_MEMORY_MIN,
• TaskManagerOptions.NETWORK_BUFFERS_MEMORY_MAX, and
• TaskManagerOptions.NETWORK_NUM_BUFFERS (fallback if the ones above do not exist)

.

Parameters:

tmConfig - task manager services configuration object

maxJvmHeapMemory - the maximum JVM heap size

Returns:

memory to use for network buffers (in bytes)

calculateHeapSizeMB

public static long calculateHeapSizeMB(long totalJavaMemorySizeMB,
                                       Configuration config)

Calculates the amount of heap memory to use (to set via -Xmx and -Xms) based on the total memory to use and the given configuration parameters.

Parameters:

totalJavaMemorySizeMB - overall available memory to use (heap and off-heap)

config - configuration object

Returns:

heap memory to use (in megabytes)