1. Deployment & Operations
2. Memory Configuration
3. Migration Guide

Migration Guide

The memory setup of task managers has changed a lot with the 1.10 release. Many configuration options were removed or their semantics changed. This guide will help you to migrate the memory configuration from Flink <= 1.9 to >= 1.10.

• Changes in Configuration Options
• Total Memory (Previously Heap Memory)
• JVM Heap Memory
• Managed Memory
  • Explicit Size
  • Fraction
  • RocksDB state
  • Other changes
• Container Cut-Off Memory
• Default Configuration in flink-conf.yaml

Note Before version 1.10, Flink did not require that memory related options are set at all as they all had default values. The new memory configuration requires that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:

• taskmanager.memory.flink.size
• taskmanager.memory.process.size
• taskmanager.memory.task.heap.size and taskmanager.memory.managed.size

The default flink-conf.yaml shipped with Flink sets taskmanager.memory.process.size to make the default memory configuration consistent.

This spreadsheet can also help to evaluate and compare the results of the legacy and new memory computations.

Changes in Configuration Options

This chapter shortly lists all changes to Flink’s memory configuration options introduced with the 1.10 release. It also references other chapters for more details about migrating to the new configuration options.

The following options are completely removed. If they are still used, they will be ignored.

The following options are deprecated but if they are still used they will be interpreted as new options for backwards compatibility:

Although, the network memory configuration has not changed too much it is recommended to verify its configuration. It can change if other memory components have new sizes, e.g. the total memory which the network can be a fraction of. See also new detailed memory model.

The container cut-off configuration options, containerized.heap-cutoff-ratio and containerized.heap-cutoff-min, have no effect for task manager processes anymore but they still have the same semantics for the job manager process. See also how to migrate container cut-off.

Total Memory (Previously Heap Memory)

The previous options which were responsible for the total memory used by Flink are taskmanager.heap.size or taskmanager.heap.mb. Despite their naming, they included not only JVM heap but also other off-heap memory components. The options have been deprecated.

The Mesos integration also had a separate option with the same semantics: mesos.resourcemanager.tasks.mem which has also been deprecated.

If the mentioned legacy options are used without specifying the corresponding new options, they will be directly translated into the following new options:

• Total Flink memory (taskmanager.memory.flink.size) for standalone deployments
• Total process memory (taskmanager.memory.process.size) for containerized deployments (Yarn or Mesos)

It is also recommended to use these new options instead of the legacy ones as they might be completely removed in the following releases.

See also how to configure total memory now.

JVM Heap Memory

JVM heap memory previously consisted of the managed memory (if configured to be on-heap) and the rest which included any other usages of heap memory. This rest was always implicitly derived as the remaining part of the total memory, see also how to migrate managed memory.

Now, if only total Flink memory or total process memory is configured, then the JVM heap is also derived as the rest of what is left after subtracting all other components from the total memory, see also how to configure total memory.

Additionally, you can now have more direct control over the JVM heap assigned to the operator tasks (taskmanager.memory.task.heap.size), see also Task (Operator) Heap Memory. The JVM heap memory is also used by the heap state backends (MemoryStateBackend or FsStateBackend) if it is chosen for streaming jobs.

A part of the JVM heap is now always reserved for Flink framework (taskmanager.memory.framework.heap.size). See also Framework memory.

Managed Memory

See also how to configure managed memory now.

Explicit Size

The previous option to configure managed memory size (taskmanager.memory.size) was renamed to taskmanager.memory.managed.size and deprecated. It is recommended to use the new option because the legacy one can be removed in future releases.

Fraction

If not set explicitly, the managed memory could be previously specified as a fraction (taskmanager.memory.fraction) of the total memory minus network memory and container cut-off (only for Yarn and Mesos deployments). This option has been completely removed and will have no effect if still used. Please, use the new option taskmanager.memory.managed.fraction instead. This new option will set the managed memory to the specified fraction of the total Flink memory if its size is not set explicitly by taskmanager.memory.managed.size.

RocksDB state

If the RocksDBStateBackend is chosen for a streaming job, its native memory consumption should now be accounted for in managed memory. The RocksDB memory allocation is limited by the managed memory size. This should prevent the killing of containers on Yarn or Mesos. You can disable the RocksDB memory control by setting state.backend.rocksdb.memory.managed to false. See also how to migrate container cut-off.

Other changes

Additionally, the following changes have been made:

• The managed memory is always off-heap now. The configuration option taskmanager.memory.off-heap is removed and will have no effect anymore.
• The managed memory now uses native memory which is not direct memory. It means that the managed memory is no longer accounted for in the JVM direct memory limit.
• The managed memory is always lazily allocated now. The configuration option taskmanager.memory.preallocate is removed and will have no effect anymore.

Container Cut-Off Memory

For containerized deployments, you could previously specify a cut-off memory. This memory could accommodate for unaccounted memory allocations. Dependencies which were not directly controlled by Flink were the main source of those allocations, e.g. RocksDB, internals of JVM, etc. This is no longer available and the related configuration options (containerized.heap-cutoff-ratio and containerized.heap-cutoff-min) will have no effect on the task manager process anymore. The new memory model introduced more specific memory components, described further, to address these concerns.

In streaming jobs which use RocksDBStateBackend, the RocksDB native memory consumption should be accounted for as a part of the managed memory now. The RocksDB memory allocation is also limited by the configured size of the managed memory. See also migrating managed memory and how to configure managed memory now.

The other direct or native off-heap memory consumers can now be addressed by the following new configuration options:

• Task off-heap memory (taskmanager.memory.task.off-heap.size)
• Framework off-heap memory (taskmanager.memory.framework.off-heap.size)
• JVM metaspace (taskmanager.memory.jvm-metaspace.size)
• JVM overhead (see also detailed new memory model)

Note The job manager still has container cut-off memory configuration options. The mentioned configuration options remain valid for the job manager in the same way as before.

Default Configuration in flink-conf.yaml

This section describes the changes of the default flink-conf.yaml shipped with Flink.

The total memory (taskmanager.heap.size) is replaced by taskmanager.memory.process.size in the default flink-conf.yaml. The value is also increased from 1024Mb to 1568Mb. See also how to configure total memory now.