IMPORTANT MemoryStateBackend is deprecated in favor of HashMapStateBackend and
JobManagerCheckpointStorage. This change does not affect the runtime characteristics of your
Jobs and is simply an API change to help better communicate the ways Flink separates local state
storage from fault tolerance. Jobs can be upgraded without loss of state. If configuring
your state backend via the StreamExecutionEnvironment please make the following changes.
If you are configuring your state backend via the flink-conf.yaml please make the following
This state backend holds the working state in the memory (JVM heap) of the TaskManagers.
The state backend checkpoints state directly to the JobManager’s memory (hence the backend’s
name), but the checkpoints will be persisted to a file system for high-availability setups and
savepoints. The MemoryStateBackend is consequently a FileSystem-based backend that can work
without a file system dependency in simple setups.
This state backend should be used only for experimentation, quick local setups,
or for streaming applications that have very small state: Because it requires checkpoints to
go through the JobManager’s memory, larger state will occupy larger portions of the
JobManager’s main memory, reducing operational stability.
For any other setup, the
FsStateBackend should be used. The
holds the working state on the TaskManagers in the same way, but checkpoints state directly to
files rather then to the JobManager’s memory, thus supporting large state sizes.
State Size Considerations
State checkpointing with this state backend is subject to the following conditions:
Each individual state must not exceed the configured maximum state size
All state from one task (i.e., the sum of all operator states and keyed states from all
chained operators of the task) must not exceed what the RPC system supports, which is
be default < 10 MB. That limit can be configured up, but that is typically not advised.
The sum of all states in the application times all retained checkpoints must comfortably
fit into the JobManager’s JVM heap space.
For the use cases where the state sizes can be handled by this backend, the backend does
guarantee persistence for savepoints, externalized checkpoints (of configured), and checkpoints
(when high-availability is configured).
As for all state backends, this backend can either be configured within the application (by
creating the backend with the respective constructor parameters and setting it on the execution
environment) or by specifying it in the Flink configuration.
If the state backend was specified in the application, it may pick up additional configuration
parameters from the Flink configuration. For example, if the backend if configured in the
application without a default savepoint directory, it will pick up a default savepoint
directory specified in the Flink configuration of the running job/cluster. That behavior is
implemented via the
Gets the maximum size that an individual state can have, as configured in the constructor (by default
Gets whether the key/value data structures are asynchronously snapshotted.