org.apache.flink.configuration

Class CheckpointingOptions

java.lang.Object

org.apache.flink.configuration.CheckpointingOptions

@PublicEvolving
public class CheckpointingOptions
extends Object

A collection of all configuration options that relate to checkpoints and savepoints.

Field Summary

Fields

Modifier and Type	Field and Description
static ConfigOption<Duration>	ALIGNED_CHECKPOINT_TIMEOUT
static ConfigOption<Boolean>	ASYNC_SNAPSHOTS Deprecated. Checkpoints are always asynchronous.
static ConfigOption<String>	CHECKPOINT_STORAGE The checkpoint storage used to checkpoint state for recovery.
static ConfigOption<CheckpointingMode>	CHECKPOINTING_CONSISTENCY_MODE
static ConfigOption<Duration>	CHECKPOINTING_INTERVAL
static ConfigOption<Duration>	CHECKPOINTING_INTERVAL_DURING_BACKLOG
static ConfigOption<Duration>	CHECKPOINTING_TIMEOUT
static ConfigOption<String>	CHECKPOINTS_DIRECTORY The default directory used for storing the data files and meta data of checkpoints in a Flink supported filesystem.
static ConfigOption<Boolean>	CLEANER_PARALLEL_MODE
static ConfigOption<Boolean>	CREATE_CHECKPOINT_SUB_DIR Whether to create sub-directories named by job id to store the data files and meta data of checkpoints.
static ConfigOption<Boolean>	ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH
static ConfigOption<Boolean>	ENABLE_UNALIGNED
static ConfigOption<Boolean>	ENABLE_UNALIGNED_INTERRUPTIBLE_TIMERS
static ConfigOption<ExternalizedCheckpointRetention>	EXTERNALIZED_CHECKPOINT_RETENTION
static ConfigOption<Boolean>	FILE_MERGING_ACROSS_BOUNDARY Whether to allow merging data of multiple checkpoints into one physical file.
static ConfigOption<Boolean>	FILE_MERGING_ENABLED Whether to enable merging multiple checkpoint files into one, which will greatly reduce the number of small checkpoint files.
static ConfigOption<MemorySize>	FILE_MERGING_MAX_FILE_SIZE The max size of a physical file for merged checkpoints.
static ConfigOption<Float>	FILE_MERGING_MAX_SPACE_AMPLIFICATION Space amplification stands for the magnification of the occupied space compared to the amount of valid data.
static ConfigOption<Integer>	FILE_MERGING_MAX_SUBTASKS_PER_FILE The upper limit of the file pool size based on the number of subtasks within each TM (only for merging private state at Task Manager level).
static ConfigOption<Boolean>	FILE_MERGING_POOL_BLOCKING Whether to use Blocking or Non-Blocking pool for merging physical files.
static ConfigOption<Boolean>	FORCE_UNALIGNED
static ConfigOption<MemorySize>	FS_SMALL_FILE_THRESHOLD The minimum size of state data files.
static ConfigOption<Integer>	FS_WRITE_BUFFER_SIZE The default size of the write buffer for the checkpoint streams that write to file systems.
static ConfigOption<Boolean>	INCREMENTAL_CHECKPOINTS Option whether to create incremental checkpoints, if possible.
static ConfigOption<Boolean>	LOCAL_BACKUP_ENABLED This option configures local backup for the state backend, which indicates whether to make backup checkpoint on local disk.
static ConfigOption<Boolean>	LOCAL_RECOVERY Deprecated. use StateRecoveryOptions.LOCAL_RECOVERY and LOCAL_BACKUP_ENABLED instead.
static ConfigOption<String>	LOCAL_RECOVERY_TASK_MANAGER_STATE_ROOT_DIRS The config parameter defining the root directories for storing file-based state for local recovery.
static ConfigOption<Integer>	MAX_CONCURRENT_CHECKPOINTS
static ConfigOption<Integer>	MAX_RETAINED_CHECKPOINTS The maximum number of completed checkpoints to retain.
static ConfigOption<Duration>	MIN_PAUSE_BETWEEN_CHECKPOINTS
static ConfigOption<String>	SAVEPOINT_DIRECTORY The default directory for savepoints.
static ConfigOption<String>	STATE_BACKEND Deprecated. Use StateBackendOptions.STATE_BACKEND.
static ConfigOption<Integer>	TOLERABLE_FAILURE_NUMBER
static ConfigOption<Integer>	UNALIGNED_MAX_SUBTASKS_PER_CHANNEL_STATE_FILE

Constructor Summary

Constructors

Constructor and Description
CheckpointingOptions()

Method Summary

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

STATE_BACKEND

@Deprecated
public static final ConfigOption<String> STATE_BACKEND

Deprecated. Use StateBackendOptions.STATE_BACKEND.

The checkpoint storage used to store operator state locally within the cluster during execution.

The implementation can be specified either via their shortcut name, or via the class name of a StateBackendFactory. If a StateBackendFactory class name is specified, the factory is instantiated (via its zero-argument constructor) and its StateBackendFactory#createFromConfig(ReadableConfig, ClassLoader) method is called.

Recognized shortcut names are 'hashmap' and 'rocksdb'.

CHECKPOINT_STORAGE

public static final ConfigOption<String> CHECKPOINT_STORAGE

The checkpoint storage used to checkpoint state for recovery.

The implementation can be specified either via their shortcut name, or via the class name of a CheckpointStorageFactory. If a CheckpointStorageFactory class name is specified, the factory is instantiated (via its zero-argument constructor) and its CheckpointStorageFactory#createFromConfig(ReadableConfig, ClassLoader) method is called.

Recognized shortcut names are 'jobmanager' and 'filesystem'.

CHECKPOINT_STORAGE and CHECKPOINTS_DIRECTORY are usually combined to configure the checkpoint location. By default, the checkpoint meta data and actual program state will be stored in the JobManager's memory directly.

When CHECKPOINT_STORAGE is set to 'jobmanager', if CHECKPOINTS_DIRECTORY is configured, the meta data of checkpoints will be persisted to the path specified by CHECKPOINTS_DIRECTORY. Otherwise, the meta data will be stored in the JobManager's memory. When CHECKPOINT_STORAGE is set to 'filesystem', a valid path must be configured to CHECKPOINTS_DIRECTORY, and the checkpoint meta data and actual program state will both be persisted to the path.

MAX_RETAINED_CHECKPOINTS

public static final ConfigOption<Integer> MAX_RETAINED_CHECKPOINTS

The maximum number of completed checkpoints to retain.

CLEANER_PARALLEL_MODE

public static final ConfigOption<Boolean> CLEANER_PARALLEL_MODE

ASYNC_SNAPSHOTS

@Deprecated
public static final ConfigOption<Boolean> ASYNC_SNAPSHOTS

Deprecated. Checkpoints are always asynchronous.

INCREMENTAL_CHECKPOINTS

public static final ConfigOption<Boolean> INCREMENTAL_CHECKPOINTS

Option whether to create incremental checkpoints, if possible. For an incremental checkpoint, only a diff from the previous checkpoint is stored, rather than the complete checkpoint state.

Once enabled, the state size shown in web UI or fetched from rest API only represents the delta checkpoint size instead of full checkpoint size.

Some state backends may not support incremental checkpoints and ignore this option.

LOCAL_RECOVERY

@Deprecated
public static final ConfigOption<Boolean> LOCAL_RECOVERY

Deprecated. use StateRecoveryOptions.LOCAL_RECOVERY and LOCAL_BACKUP_ENABLED instead.

This option configures local recovery for this state backend. By default, local recovery is deactivated.

Local recovery currently only covers keyed state backends (including both the EmbeddedRocksDBStateBackend and the HashMapStateBackend).

LOCAL_RECOVERY_TASK_MANAGER_STATE_ROOT_DIRS

public static final ConfigOption<String> LOCAL_RECOVERY_TASK_MANAGER_STATE_ROOT_DIRS

The config parameter defining the root directories for storing file-based state for local recovery.

Local recovery currently only covers keyed state backends. Currently, MemoryStateBackend does not support local recovery and ignore this option.

SAVEPOINT_DIRECTORY

public static final ConfigOption<String> SAVEPOINT_DIRECTORY

The default directory for savepoints. Used by the state backends that write savepoints to file systems (HashMapStateBackend, EmbeddedRocksDBStateBackend).

CHECKPOINTS_DIRECTORY

public static final ConfigOption<String> CHECKPOINTS_DIRECTORY

The default directory used for storing the data files and meta data of checkpoints in a Flink supported filesystem. The storage path must be accessible from all participating processes/nodes(i.e. all TaskManagers and JobManagers). If CHECKPOINT_STORAGE is set to 'jobmanager', only the meta data of checkpoints will be stored in this directory.

CREATE_CHECKPOINT_SUB_DIR

public static final ConfigOption<Boolean> CREATE_CHECKPOINT_SUB_DIR

Whether to create sub-directories named by job id to store the data files and meta data of checkpoints. The default value is true to enable user could run several jobs with the same checkpoint directory at the same time. If this value is set to false, pay attention not to run several jobs with the same directory simultaneously.

FS_SMALL_FILE_THRESHOLD

public static final ConfigOption<MemorySize> FS_SMALL_FILE_THRESHOLD

The minimum size of state data files. All state chunks smaller than that are stored inline in the root checkpoint metadata file.

FS_WRITE_BUFFER_SIZE

public static final ConfigOption<Integer> FS_WRITE_BUFFER_SIZE

The default size of the write buffer for the checkpoint streams that write to file systems.

LOCAL_BACKUP_ENABLED

public static final ConfigOption<Boolean> LOCAL_BACKUP_ENABLED

This option configures local backup for the state backend, which indicates whether to make backup checkpoint on local disk. If not configured, fallback to StateRecoveryOptions.LOCAL_RECOVERY. By default, local backup is deactivated. Local backup currently only covers keyed state backends (including both the EmbeddedRocksDBStateBackend and the HashMapStateBackend).

FILE_MERGING_ENABLED

@Experimental
public static final ConfigOption<Boolean> FILE_MERGING_ENABLED

Whether to enable merging multiple checkpoint files into one, which will greatly reduce the number of small checkpoint files. See FLIP-306 for details.

Note: This is an experimental feature under evaluation, make sure you're aware of the possible effects of enabling it.

FILE_MERGING_ACROSS_BOUNDARY

@Experimental
public static final ConfigOption<Boolean> FILE_MERGING_ACROSS_BOUNDARY

Whether to allow merging data of multiple checkpoints into one physical file. If this option is set to false, only merge files within checkpoint boundaries. Otherwise, it is possible for the logical files of different checkpoints to share the same physical file.

FILE_MERGING_MAX_FILE_SIZE

@Experimental
public static final ConfigOption<MemorySize> FILE_MERGING_MAX_FILE_SIZE

The max size of a physical file for merged checkpoints.

FILE_MERGING_POOL_BLOCKING

@Experimental
public static final ConfigOption<Boolean> FILE_MERGING_POOL_BLOCKING

Whether to use Blocking or Non-Blocking pool for merging physical files. A Non-Blocking pool will always provide usable physical file without blocking. It may create many physical files if poll file frequently. When poll a small file from a Blocking pool, it may be blocked until the file is returned.

FILE_MERGING_MAX_SUBTASKS_PER_FILE

@Experimental
public static final ConfigOption<Integer> FILE_MERGING_MAX_SUBTASKS_PER_FILE

The upper limit of the file pool size based on the number of subtasks within each TM (only for merging private state at Task Manager level).

TODO: remove '@Documentation.ExcludeFromDocumentation' after the feature is implemented.

FILE_MERGING_MAX_SPACE_AMPLIFICATION

@Experimental
public static final ConfigOption<Float> FILE_MERGING_MAX_SPACE_AMPLIFICATION

Space amplification stands for the magnification of the occupied space compared to the amount of valid data. The more space amplification is, the more waste of space will be. This configs a space amplification above which a re-uploading for physical files will be triggered to reclaim space.

CHECKPOINTING_CONSISTENCY_MODE

public static final ConfigOption<CheckpointingMode> CHECKPOINTING_CONSISTENCY_MODE

CHECKPOINTING_TIMEOUT

public static final ConfigOption<Duration> CHECKPOINTING_TIMEOUT

MAX_CONCURRENT_CHECKPOINTS

public static final ConfigOption<Integer> MAX_CONCURRENT_CHECKPOINTS

MIN_PAUSE_BETWEEN_CHECKPOINTS

public static final ConfigOption<Duration> MIN_PAUSE_BETWEEN_CHECKPOINTS

TOLERABLE_FAILURE_NUMBER

public static final ConfigOption<Integer> TOLERABLE_FAILURE_NUMBER

EXTERNALIZED_CHECKPOINT_RETENTION

public static final ConfigOption<ExternalizedCheckpointRetention> EXTERNALIZED_CHECKPOINT_RETENTION

CHECKPOINTING_INTERVAL_DURING_BACKLOG

public static final ConfigOption<Duration> CHECKPOINTING_INTERVAL_DURING_BACKLOG

CHECKPOINTING_INTERVAL

public static final ConfigOption<Duration> CHECKPOINTING_INTERVAL

ENABLE_UNALIGNED

public static final ConfigOption<Boolean> ENABLE_UNALIGNED

ALIGNED_CHECKPOINT_TIMEOUT

public static final ConfigOption<Duration> ALIGNED_CHECKPOINT_TIMEOUT

FORCE_UNALIGNED

public static final ConfigOption<Boolean> FORCE_UNALIGNED

ENABLE_UNALIGNED_INTERRUPTIBLE_TIMERS

@Experimental
public static final ConfigOption<Boolean> ENABLE_UNALIGNED_INTERRUPTIBLE_TIMERS

ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH

public static final ConfigOption<Boolean> ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH

UNALIGNED_MAX_SUBTASKS_PER_CHANNEL_STATE_FILE

public static final ConfigOption<Integer> UNALIGNED_MAX_SUBTASKS_PER_CHANNEL_STATE_FILE

Constructor Detail

CheckpointingOptions

public CheckpointingOptions()