public interface FileMergingSnapshotManager extends Closeable
TODO (FLINK-32075): leverage checkpoint notification to delete logical files.
Modifier and Type | Interface and Description |
---|---|
static class |
FileMergingSnapshotManager.SubtaskKey
A key identifies a subtask.
|
void initFileSystem(FileSystem fileSystem, Path checkpointBaseDir, Path sharedStateDir, Path taskOwnedStateDir, int writeBufferSize) throws IllegalArgumentException
The layout of checkpoint directory: /user-defined-checkpoint-dir /{job-id} (checkpointBaseDir) | + --shared/ | + --subtask-1/ + -- merged shared state files + --subtask-2/ + -- merged shared state files + --taskowned/ + -- merged private state files + --chk-1/ + --chk-2/ + --chk-3/
The reason why initializing directories in this method instead of the constructor is that
the FileMergingSnapshotManager itself belongs to the TaskStateManager
, which is
initialized when receiving a task, while the base directories for checkpoint are created by
FsCheckpointStorageAccess
when the state backend initializes per subtask. After the
checkpoint directories are initialized, the managed subdirectories are initialized here.
Note: This method may be called several times, the implementation should ensure
idempotency, and throw IllegalArgumentException
when any of the path in params change
across function calls.
fileSystem
- The filesystem to write to.checkpointBaseDir
- The base directory for checkpoints.sharedStateDir
- The directory for shared checkpoint data.taskOwnedStateDir
- The name of the directory for state not owned/released by the
master, but by the TaskManagers.writeBufferSize
- The buffer size for writing files to the file system.IllegalArgumentException
- thrown if these three paths are not deterministic across
calls.void registerSubtaskForSharedStates(FileMergingSnapshotManager.SubtaskKey subtaskKey)
subtaskKey
- the subtask key identifying a subtask.for layout information.
FileMergingCheckpointStateOutputStream createCheckpointStateOutputStream(FileMergingSnapshotManager.SubtaskKey subtaskKey, long checkpointId, CheckpointedStateScope scope)
FileMergingCheckpointStateOutputStream
. According to the file merging
strategy, the streams returned by multiple calls to this function may share the same
underlying physical file, and each stream writes to a segment of the physical file.subtaskKey
- The subtask key identifying the subtask.checkpointId
- ID of the checkpoint.scope
- The state's scope, whether it is exclusive or shared.Path getManagedDir(FileMergingSnapshotManager.SubtaskKey subtaskKey, CheckpointedStateScope scope)
initFileSystem(org.apache.flink.core.fs.FileSystem, org.apache.flink.core.fs.Path, org.apache.flink.core.fs.Path, org.apache.flink.core.fs.Path, int)
or registerSubtaskForSharedStates(org.apache.flink.runtime.checkpoint.filemerging.FileMergingSnapshotManager.SubtaskKey)
.subtaskKey
- the subtask key identifying the subtask.scope
- the checkpoint scope.Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.