1. Deployment & Operations
2. Checkpoints

Checkpoints

• Overview
• Externalized Checkpoints

Overview

TBD

Externalized Checkpoints

You can configure periodic checkpoints to be persisted externally. Externalized checkpoints write their meta data out to persistent storage and are not automatically cleaned up when the job fails. This way, you will have a checkpoint around to resume from if your job fails.

CheckpointConfig config = env.getCheckpointConfig();
config.enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);

The ExternalizedCheckpointCleanup mode configures what happens with externalized checkpoints when you cancel the job:

• ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION: Retain the externalized checkpoint when the job is cancelled. Note that you have to manually clean up the checkpoint state after cancellation in this case.

• ExternalizedCheckpointCleanup.DELETE_ON_CANCELLATION: Delete the externalized checkpoint when the job is cancelled. The checkpoint state will only be available if the job fails.

The target directory for the checkpoint is determined from the default checkpoint directory configuration. This is configured via the configuration key state.checkpoints.dir, which should point to the desired target directory:

state.checkpoints.dir: hdfs:///checkpoints/

This directory will then contain the checkpoint meta data required to restore the checkpoint. The actual checkpoint files will still be available in their configured directory. You currently can only set this via the configuration files.

Follow the savepoint guide when you want to resume from a specific checkpoint.