public class DefaultCompletedCheckpointStore<R extends ResourceVersion<R>> extends AbstractCompleteCheckpointStore
CompletedCheckpointStore
. Combined with different StateHandleStore
, we could persist the completed
checkpoints to various storage.
During recovery, the latest checkpoint is read from StateHandleStore
. If there is more
than one, only the latest one is used and older ones are discarded (even if the maximum number of
retained checkpoints is greater than one).
If there is a network partition and multiple JobManagers run concurrent checkpoints for the same program, it is OK to take any valid successful checkpoint as long as the "history" of checkpoints is consistent. Currently, after recovery we start out with only a single checkpoint to circumvent those situations.
Constructor and Description |
---|
DefaultCompletedCheckpointStore(int maxNumberOfCheckpointsToRetain,
StateHandleStore<CompletedCheckpoint,R> stateHandleStore,
CheckpointStoreUtil completedCheckpointStoreUtil,
Collection<CompletedCheckpoint> completedCheckpoints,
SharedStateRegistry sharedStateRegistry,
Executor executor)
Creates a
DefaultCompletedCheckpointStore instance. |
Modifier and Type | Method and Description |
---|---|
CompletedCheckpoint |
addCheckpointAndSubsumeOldestOne(CompletedCheckpoint checkpoint,
CheckpointsCleaner checkpointsCleaner,
Runnable postCleanup)
Synchronously writes the new checkpoints to state handle store and asynchronously removes
older ones.
|
List<CompletedCheckpoint> |
getAllCheckpoints()
Returns all
CompletedCheckpoint instances. |
int |
getMaxNumberOfRetainedCheckpoints()
Returns the max number of retained checkpoints.
|
int |
getNumberOfRetainedCheckpoints()
Returns the current number of retained checkpoints.
|
boolean |
requiresExternalizedCheckpoints()
This method returns whether the completed checkpoint store requires checkpoints to be
externalized.
|
void |
shutdown(JobStatus jobStatus,
CheckpointsCleaner checkpointsCleaner)
Shuts down the store.
|
findLowest, getSharedStateRegistry, unregisterUnusedState
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
getLatestCheckpoint, getLatestCheckpointId
public DefaultCompletedCheckpointStore(int maxNumberOfCheckpointsToRetain, StateHandleStore<CompletedCheckpoint,R> stateHandleStore, CheckpointStoreUtil completedCheckpointStoreUtil, Collection<CompletedCheckpoint> completedCheckpoints, SharedStateRegistry sharedStateRegistry, Executor executor)
DefaultCompletedCheckpointStore
instance.maxNumberOfCheckpointsToRetain
- The maximum number of checkpoints to retain (at least
1). Adding more checkpoints than this results in older checkpoints being discarded. On
recovery, we will only start with a single checkpoint.stateHandleStore
- Completed checkpoints in external storecompletedCheckpointStoreUtil
- utilities for completed checkpoint storeexecutor
- to execute blocking callspublic boolean requiresExternalizedCheckpoints()
CompletedCheckpointStore
public CompletedCheckpoint addCheckpointAndSubsumeOldestOne(CompletedCheckpoint checkpoint, CheckpointsCleaner checkpointsCleaner, Runnable postCleanup) throws Exception
checkpoint
- Completed checkpoint to add.PossibleInconsistentStateException
- if adding the checkpoint failed and leaving the
system in a possibly inconsistent state, i.e. it's uncertain whether the checkpoint
metadata was fully written to the underlying systems or not.Exception
public List<CompletedCheckpoint> getAllCheckpoints()
CompletedCheckpointStore
CompletedCheckpoint
instances.
Returns an empty list if no checkpoint has been added yet.
public int getNumberOfRetainedCheckpoints()
CompletedCheckpointStore
public int getMaxNumberOfRetainedCheckpoints()
CompletedCheckpointStore
public void shutdown(JobStatus jobStatus, CheckpointsCleaner checkpointsCleaner) throws Exception
CompletedCheckpointStore
The job status is forwarded and used to decide whether state should actually be discarded
or kept. SharedStateRegistry.unregisterUnusedState(long)
and CheckpointsCleaner.cleanSubsumedCheckpoints(long, java.util.Set<java.lang.Long>, java.lang.Runnable, java.util.concurrent.Executor)
should be called here to subsume unused state.
shutdown
in interface CompletedCheckpointStore
shutdown
in class AbstractCompleteCheckpointStore
jobStatus
- Job state on shut downcheckpointsCleaner
- that will cleanup completed checkpoints if neededException
Copyright © 2014–2024 The Apache Software Foundation. All rights reserved.