Class CheckpointingOptions
- java.lang.Object
-
- org.apache.flink.configuration.CheckpointingOptions
-
@PublicEvolving public class CheckpointingOptions extends Object
A collection of all configuration options that relate to checkpoints and savepoints.
-
-
Field Summary
Fields Modifier and Type Field Description static ConfigOption<Duration>
ALIGNED_CHECKPOINT_TIMEOUT
static ConfigOption<String>
CHECKPOINT_STORAGE
The checkpoint storage used to checkpoint state for recovery.static ConfigOption<CheckpointingMode>
CHECKPOINTING_CONSISTENCY_MODE
static ConfigOption<Duration>
CHECKPOINTING_INTERVAL
static ConfigOption<Duration>
CHECKPOINTING_INTERVAL_DURING_BACKLOG
static ConfigOption<Duration>
CHECKPOINTING_TIMEOUT
static ConfigOption<String>
CHECKPOINTS_DIRECTORY
The default directory used for storing the data files and meta data of checkpoints in a Flink supported filesystem.static ConfigOption<Boolean>
CLEANER_PARALLEL_MODE
static ConfigOption<Boolean>
CREATE_CHECKPOINT_SUB_DIR
Whether to create sub-directories named by job id to store the data files and meta data of checkpoints.static ConfigOption<Boolean>
ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH
static ConfigOption<Boolean>
ENABLE_UNALIGNED
static ConfigOption<Boolean>
ENABLE_UNALIGNED_INTERRUPTIBLE_TIMERS
static ConfigOption<ExternalizedCheckpointRetention>
EXTERNALIZED_CHECKPOINT_RETENTION
static ConfigOption<Boolean>
FILE_MERGING_ACROSS_BOUNDARY
Whether to allow merging data of multiple checkpoints into one physical file.static ConfigOption<Boolean>
FILE_MERGING_ENABLED
Whether to enable merging multiple checkpoint files into one, which will greatly reduce the number of small checkpoint files.static ConfigOption<MemorySize>
FILE_MERGING_MAX_FILE_SIZE
The max size of a physical file for merged checkpoints.static ConfigOption<Float>
FILE_MERGING_MAX_SPACE_AMPLIFICATION
Space amplification stands for the magnification of the occupied space compared to the amount of valid data.static ConfigOption<Integer>
FILE_MERGING_MAX_SUBTASKS_PER_FILE
The upper limit of the file pool size based on the number of subtasks within each TM (only for merging private state at Task Manager level).static ConfigOption<Boolean>
FILE_MERGING_POOL_BLOCKING
Whether to use Blocking or Non-Blocking pool for merging physical files.static ConfigOption<Boolean>
FORCE_UNALIGNED
static ConfigOption<MemorySize>
FS_SMALL_FILE_THRESHOLD
The minimum size of state data files.static ConfigOption<Integer>
FS_WRITE_BUFFER_SIZE
The default size of the write buffer for the checkpoint streams that write to file systems.static ConfigOption<Boolean>
INCREMENTAL_CHECKPOINTS
Option whether to create incremental checkpoints, if possible.static ConfigOption<Boolean>
LOCAL_BACKUP_ENABLED
This option configures local backup for the state backend, which indicates whether to make backup checkpoint on local disk.static ConfigOption<String>
LOCAL_RECOVERY_TASK_MANAGER_STATE_ROOT_DIRS
The config parameter defining the root directories for storing file-based state for local recovery.static ConfigOption<Integer>
MAX_CONCURRENT_CHECKPOINTS
static ConfigOption<Integer>
MAX_RETAINED_CHECKPOINTS
The maximum number of completed checkpoints to retain.static ConfigOption<Duration>
MIN_PAUSE_BETWEEN_CHECKPOINTS
static ConfigOption<String>
SAVEPOINT_DIRECTORY
The default directory for savepoints.static ConfigOption<Integer>
TOLERABLE_FAILURE_NUMBER
static ConfigOption<Integer>
UNALIGNED_MAX_SUBTASKS_PER_CHANNEL_STATE_FILE
-
Constructor Summary
Constructors Constructor Description CheckpointingOptions()
-
-
-
Field Detail
-
CHECKPOINT_STORAGE
public static final ConfigOption<String> CHECKPOINT_STORAGE
The checkpoint storage used to checkpoint state for recovery.The implementation can be specified either via their shortcut name, or via the class name of a
CheckpointStorageFactory
. If a CheckpointStorageFactory class name is specified, the factory is instantiated (via its zero-argument constructor) and itsCheckpointStorageFactory#createFromConfig(ReadableConfig, ClassLoader)
method is called.Recognized shortcut names are 'jobmanager' and 'filesystem'.
CHECKPOINT_STORAGE
andCHECKPOINTS_DIRECTORY
are usually combined to configure the checkpoint location. By default, the checkpoint meta data and actual program state will be stored in the JobManager's memory directly.- When
CHECKPOINT_STORAGE
is set to 'jobmanager', ifCHECKPOINTS_DIRECTORY
is configured, the meta data of checkpoints will be persisted to the path specified byCHECKPOINTS_DIRECTORY
. Otherwise, the meta data will be stored in the JobManager's memory.- When
CHECKPOINT_STORAGE
is set to 'filesystem', a valid path must be configured toCHECKPOINTS_DIRECTORY
, and the checkpoint meta data and actual program state will both be persisted to the path. - When
-
MAX_RETAINED_CHECKPOINTS
public static final ConfigOption<Integer> MAX_RETAINED_CHECKPOINTS
The maximum number of completed checkpoints to retain.
-
CLEANER_PARALLEL_MODE
public static final ConfigOption<Boolean> CLEANER_PARALLEL_MODE
-
INCREMENTAL_CHECKPOINTS
public static final ConfigOption<Boolean> INCREMENTAL_CHECKPOINTS
Option whether to create incremental checkpoints, if possible. For an incremental checkpoint, only a diff from the previous checkpoint is stored, rather than the complete checkpoint state.Once enabled, the state size shown in web UI or fetched from rest API only represents the delta checkpoint size instead of full checkpoint size.
Some state backends may not support incremental checkpoints and ignore this option.
-
LOCAL_RECOVERY_TASK_MANAGER_STATE_ROOT_DIRS
public static final ConfigOption<String> LOCAL_RECOVERY_TASK_MANAGER_STATE_ROOT_DIRS
The config parameter defining the root directories for storing file-based state for local recovery.Local recovery currently only covers keyed state backends. Currently, MemoryStateBackend does not support local recovery and ignore this option.
-
SAVEPOINT_DIRECTORY
public static final ConfigOption<String> SAVEPOINT_DIRECTORY
The default directory for savepoints. Used by the state backends that write savepoints to file systems (HashMapStateBackend, EmbeddedRocksDBStateBackend).
-
CHECKPOINTS_DIRECTORY
public static final ConfigOption<String> CHECKPOINTS_DIRECTORY
The default directory used for storing the data files and meta data of checkpoints in a Flink supported filesystem. The storage path must be accessible from all participating processes/nodes(i.e. all TaskManagers and JobManagers). IfCHECKPOINT_STORAGE
is set to 'jobmanager', only the meta data of checkpoints will be stored in this directory.
-
CREATE_CHECKPOINT_SUB_DIR
public static final ConfigOption<Boolean> CREATE_CHECKPOINT_SUB_DIR
Whether to create sub-directories named by job id to store the data files and meta data of checkpoints. The default value is true to enable user could run several jobs with the same checkpoint directory at the same time. If this value is set to false, pay attention not to run several jobs with the same directory simultaneously.
-
FS_SMALL_FILE_THRESHOLD
public static final ConfigOption<MemorySize> FS_SMALL_FILE_THRESHOLD
The minimum size of state data files. All state chunks smaller than that are stored inline in the root checkpoint metadata file.
-
FS_WRITE_BUFFER_SIZE
public static final ConfigOption<Integer> FS_WRITE_BUFFER_SIZE
The default size of the write buffer for the checkpoint streams that write to file systems.
-
LOCAL_BACKUP_ENABLED
public static final ConfigOption<Boolean> LOCAL_BACKUP_ENABLED
This option configures local backup for the state backend, which indicates whether to make backup checkpoint on local disk. If not configured, fallback toStateRecoveryOptions.LOCAL_RECOVERY
. By default, local backup is deactivated. Local backup currently only covers keyed state backends (including both the EmbeddedRocksDBStateBackend and the HashMapStateBackend).
-
FILE_MERGING_ENABLED
@Experimental public static final ConfigOption<Boolean> FILE_MERGING_ENABLED
Whether to enable merging multiple checkpoint files into one, which will greatly reduce the number of small checkpoint files. See FLIP-306 for details.Note: This is an experimental feature under evaluation, make sure you're aware of the possible effects of enabling it.
-
FILE_MERGING_ACROSS_BOUNDARY
@Experimental public static final ConfigOption<Boolean> FILE_MERGING_ACROSS_BOUNDARY
Whether to allow merging data of multiple checkpoints into one physical file. If this option is set to false, only merge files within checkpoint boundaries. Otherwise, it is possible for the logical files of different checkpoints to share the same physical file.
-
FILE_MERGING_MAX_FILE_SIZE
@Experimental public static final ConfigOption<MemorySize> FILE_MERGING_MAX_FILE_SIZE
The max size of a physical file for merged checkpoints.
-
FILE_MERGING_POOL_BLOCKING
@Experimental public static final ConfigOption<Boolean> FILE_MERGING_POOL_BLOCKING
Whether to use Blocking or Non-Blocking pool for merging physical files. A Non-Blocking pool will always provide usable physical file without blocking. It may create many physical files if poll file frequently. When poll a small file from a Blocking pool, it may be blocked until the file is returned.
-
FILE_MERGING_MAX_SUBTASKS_PER_FILE
@Experimental public static final ConfigOption<Integer> FILE_MERGING_MAX_SUBTASKS_PER_FILE
The upper limit of the file pool size based on the number of subtasks within each TM (only for merging private state at Task Manager level).TODO: remove '@Documentation.ExcludeFromDocumentation' after the feature is implemented.
-
FILE_MERGING_MAX_SPACE_AMPLIFICATION
@Experimental public static final ConfigOption<Float> FILE_MERGING_MAX_SPACE_AMPLIFICATION
Space amplification stands for the magnification of the occupied space compared to the amount of valid data. The more space amplification is, the more waste of space will be. This configs a space amplification above which a re-uploading for physical files will be triggered to reclaim space.
-
CHECKPOINTING_CONSISTENCY_MODE
public static final ConfigOption<CheckpointingMode> CHECKPOINTING_CONSISTENCY_MODE
-
CHECKPOINTING_TIMEOUT
public static final ConfigOption<Duration> CHECKPOINTING_TIMEOUT
-
MAX_CONCURRENT_CHECKPOINTS
public static final ConfigOption<Integer> MAX_CONCURRENT_CHECKPOINTS
-
MIN_PAUSE_BETWEEN_CHECKPOINTS
public static final ConfigOption<Duration> MIN_PAUSE_BETWEEN_CHECKPOINTS
-
TOLERABLE_FAILURE_NUMBER
public static final ConfigOption<Integer> TOLERABLE_FAILURE_NUMBER
-
EXTERNALIZED_CHECKPOINT_RETENTION
public static final ConfigOption<ExternalizedCheckpointRetention> EXTERNALIZED_CHECKPOINT_RETENTION
-
CHECKPOINTING_INTERVAL_DURING_BACKLOG
public static final ConfigOption<Duration> CHECKPOINTING_INTERVAL_DURING_BACKLOG
-
CHECKPOINTING_INTERVAL
public static final ConfigOption<Duration> CHECKPOINTING_INTERVAL
-
ENABLE_UNALIGNED
public static final ConfigOption<Boolean> ENABLE_UNALIGNED
-
ALIGNED_CHECKPOINT_TIMEOUT
public static final ConfigOption<Duration> ALIGNED_CHECKPOINT_TIMEOUT
-
FORCE_UNALIGNED
public static final ConfigOption<Boolean> FORCE_UNALIGNED
-
ENABLE_UNALIGNED_INTERRUPTIBLE_TIMERS
@Experimental public static final ConfigOption<Boolean> ENABLE_UNALIGNED_INTERRUPTIBLE_TIMERS
-
ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH
public static final ConfigOption<Boolean> ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH
-
UNALIGNED_MAX_SUBTASKS_PER_CHANNEL_STATE_FILE
public static final ConfigOption<Integer> UNALIGNED_MAX_SUBTASKS_PER_CHANNEL_STATE_FILE
-
-