public class FsCheckpointStorageAccess extends AbstractFsCheckpointStorageAccess
CHECKPOINT_DIR_PREFIX, CHECKPOINT_SHARED_STATE_DIR, CHECKPOINT_TASK_OWNED_STATE_DIR, METADATA_FILE_NAME
Constructor and Description |
---|
FsCheckpointStorageAccess(FileSystem fs,
Path checkpointBaseDirectory,
Path defaultSavepointDirectory,
JobID jobId,
int fileSizeThreshold,
int writeBufferSize) |
FsCheckpointStorageAccess(Path checkpointBaseDirectory,
Path defaultSavepointDirectory,
JobID jobId,
int fileSizeThreshold,
int writeBufferSize) |
Modifier and Type | Method and Description |
---|---|
protected CheckpointStorageLocation |
createSavepointLocation(FileSystem fs,
Path location) |
CheckpointStateToolset |
createTaskOwnedCheckpointStateToolset()
A complementary method to
CheckpointStorageWorkerView.createTaskOwnedStateStream() . |
CheckpointStateOutputStream |
createTaskOwnedStateStream()
Opens a stream to persist checkpoint state data that is owned strictly by tasks and not
attached to the life cycle of a specific checkpoint.
|
void |
initializeBaseLocationsForCheckpoint()
Initializes the necessary prerequisites for storage locations of checkpoints.
|
CheckpointStorageLocation |
initializeLocationForCheckpoint(long checkpointId)
Initializes a storage location for new checkpoint with the given ID.
|
CheckpointStreamFactory |
resolveCheckpointStorageLocation(long checkpointId,
CheckpointStorageLocationReference reference)
Resolves a storage location reference into a CheckpointStreamFactory.
|
boolean |
supportsHighlyAvailableStorage()
Checks whether this backend supports highly available storage of data.
|
createCheckpointDirectory, decodePathFromReference, encodePathAsReference, getCheckpointDirectoryForJob, getDefaultSavepointDirectory, hasDefaultSavepointLocation, initializeLocationForSavepoint, resolveCheckpoint, resolveCheckpointPointer
public FsCheckpointStorageAccess(Path checkpointBaseDirectory, @Nullable Path defaultSavepointDirectory, JobID jobId, int fileSizeThreshold, int writeBufferSize) throws IOException
IOException
public FsCheckpointStorageAccess(FileSystem fs, Path checkpointBaseDirectory, @Nullable Path defaultSavepointDirectory, JobID jobId, int fileSizeThreshold, int writeBufferSize) throws IOException
IOException
public boolean supportsHighlyAvailableStorage()
CheckpointStorageCoordinatorView
Some state backends may not support highly-available durable storage, with default settings, which makes them suitable for zero-config prototyping, but not for actual production setups.
public void initializeBaseLocationsForCheckpoint() throws IOException
CheckpointStorageCoordinatorView
For file-based checkpoint storage, this method would initialize essential base checkpoint
directories on checkpoint coordinator side and should be executed before calling CheckpointStorageCoordinatorView.initializeLocationForCheckpoint(long)
.
IOException
- Thrown, if these base storage locations cannot be initialized due to an
I/O exception.public CheckpointStorageLocation initializeLocationForCheckpoint(long checkpointId) throws IOException
CheckpointStorageCoordinatorView
The returned storage location can be used to write the checkpoint data and metadata to and to obtain the pointers for the location(s) where the actual checkpoint data should be stored.
checkpointId
- The ID (logical timestamp) of the checkpoint that should be persisted.IOException
- Thrown if the storage location cannot be initialized due to an I/O
exception.public CheckpointStreamFactory resolveCheckpointStorageLocation(long checkpointId, CheckpointStorageLocationReference reference) throws IOException
CheckpointStorageWorkerView
The reference may be the default reference
, in which case the method should return the default location, taking
existing configuration and checkpoint ID into account.
checkpointId
- The ID of the checkpoint that the location is initialized for.reference
- The checkpoint location reference.IOException
- Thrown, if the storage location cannot be initialized from the reference.public CheckpointStateOutputStream createTaskOwnedStateStream()
CheckpointStorageWorkerView
This method should be used when the persisted data cannot be immediately dropped once the checkpoint that created it is dropped. Examples are write-ahead-logs. For those, the state can only be dropped once the data has been moved to the target system, which may sometimes take longer than one checkpoint (if the target system is temporarily unable to keep up).
The fact that the job manager does not own the life cycle of this type of state means also that it is strictly the responsibility of the tasks to handle the cleanup of this data.
Developer note: In the future, we may be able to make this a special case of "shared state", where the task re-emits the shared state reference as long as it needs to hold onto the persisted state data.
public CheckpointStateToolset createTaskOwnedCheckpointStateToolset()
CheckpointStorageWorkerView
CheckpointStorageWorkerView.createTaskOwnedStateStream()
. Creates a toolset that gives
access to additional operations that can be performed in the task owned state location.protected CheckpointStorageLocation createSavepointLocation(FileSystem fs, Path location)
createSavepointLocation
in class AbstractFsCheckpointStorageAccess
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.