org.apache.flink.runtime.checkpoint.filemerging

Class WithinCheckpointFileMergingSnapshotManager

java.lang.Object

org.apache.flink.runtime.checkpoint.filemerging.FileMergingSnapshotManagerBase

org.apache.flink.runtime.checkpoint.filemerging.WithinCheckpointFileMergingSnapshotManager

All Implemented Interfaces:

Closeable, AutoCloseable, FileMergingSnapshotManager

public class WithinCheckpointFileMergingSnapshotManager
extends FileMergingSnapshotManagerBase

A FileMergingSnapshotManager that merging files within a checkpoint.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.runtime.checkpoint.filemerging.FileMergingSnapshotManager

FileMergingSnapshotManager.SubtaskKey

Field Summary

Fields inherited from class org.apache.flink.runtime.checkpoint.filemerging.FileMergingSnapshotManagerBase

checkpointDir, fs, ioExecutor, managedExclusiveStateDir, physicalFileDeleter, sharedStateDir, shouldSyncAfterClosingLogicalFile, taskOwnedStateDir

Constructor Summary

Constructors

Constructor and Description
WithinCheckpointFileMergingSnapshotManager(String id, Executor ioExecutor)

Method Summary

Modifier and Type	Method and Description
protected PhysicalFile	getOrCreatePhysicalFileForCheckpoint(FileMergingSnapshotManager.SubtaskKey subtaskKey, long checkpointId, CheckpointedStateScope scope) Get a reused physical file or create one.
protected void	returnPhysicalFileForNextReuse(FileMergingSnapshotManager.SubtaskKey subtaskKey, long checkpointId, PhysicalFile physicalFile) Try to return an existing physical file to the manager for next reuse.

Methods inherited from class org.apache.flink.runtime.checkpoint.filemerging.FileMergingSnapshotManagerBase

close, createLogicalFile, createPhysicalFile, deletePhysicalFile, generatePhysicalFilePath, getManagedDir, initFileSystem, registerSubtaskForSharedStates

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WithinCheckpointFileMergingSnapshotManager

public WithinCheckpointFileMergingSnapshotManager(String id,
                                                  Executor ioExecutor)

Method Detail

getOrCreatePhysicalFileForCheckpoint

@Nonnull
protected PhysicalFile getOrCreatePhysicalFileForCheckpoint(FileMergingSnapshotManager.SubtaskKey subtaskKey,
                                                                     long checkpointId,
                                                                     CheckpointedStateScope scope)
                                                              throws IOException

Description copied from class: FileMergingSnapshotManagerBase

Get a reused physical file or create one. This will be called in checkpoint output stream creation logic.

TODO (FLINK-32073): Implement a CheckpointStreamFactory for file-merging that uses this method to create or reuse physical files.

Basic logic of file reusing: whenever a physical file is needed, this method is called with necessary information provided for acquiring a file. The file will not be reused until it is written and returned to the reused pool by calling FileMergingSnapshotManagerBase.returnPhysicalFileForNextReuse(org.apache.flink.runtime.checkpoint.filemerging.FileMergingSnapshotManager.SubtaskKey, long, org.apache.flink.runtime.checkpoint.filemerging.PhysicalFile).

Specified by:

getOrCreatePhysicalFileForCheckpoint in class FileMergingSnapshotManagerBase

Parameters:

subtaskKey - the subtask key for the caller

checkpointId - the checkpoint id

scope - checkpoint scope

Returns:

the requested physical file.

Throws:

IOException - thrown if anything goes wrong with file system.

returnPhysicalFileForNextReuse

protected void returnPhysicalFileForNextReuse(FileMergingSnapshotManager.SubtaskKey subtaskKey,
                                              long checkpointId,
                                              PhysicalFile physicalFile)
                                       throws IOException

Description copied from class: FileMergingSnapshotManagerBase

Try to return an existing physical file to the manager for next reuse. If this physical file is no longer needed (for reusing), it will be closed.

Basic logic of file reusing, see FileMergingSnapshotManagerBase.getOrCreatePhysicalFileForCheckpoint(org.apache.flink.runtime.checkpoint.filemerging.FileMergingSnapshotManager.SubtaskKey, long, org.apache.flink.runtime.state.CheckpointedStateScope).

Specified by:

returnPhysicalFileForNextReuse in class FileMergingSnapshotManagerBase

Parameters:

subtaskKey - the subtask key for the caller

checkpointId - in which checkpoint this physical file is requested.

physicalFile - the returning checkpoint

Throws:

IOException - thrown if anything goes wrong with file system.

See Also:

#getOrCreatePhysicalFileForCheckpoint(SubtaskKey, long, CheckpointedStateScope)