public class ZooKeeperCompletedCheckpointStore extends Object implements CompletedCheckpointStore
CompletedCheckpointStore
for JobManagers running in HighAvailabilityMode.ZOOKEEPER
.
Checkpoints are added under a ZNode per job:
+----O /flink/checkpoints/<job-id> [persistent] . | . +----O /flink/checkpoints/<job-id>/1 [persistent] . . . . . . . . . . +----O /flink/checkpoints/<job-id>/N [persistent]
During recovery, the latest checkpoint is read from ZooKeeper. If there is more than one, only the latest one is used and older ones are discarded (even if the maximum number of retained checkpoints is greater than one).
If there is a network partition and multiple JobManagers run concurrent checkpoints for the same program, it is OK to take any valid successful checkpoint as long as the "history" of checkpoints is consistent. Currently, after recovery we start out with only a single checkpoint to circumvent those situations.
Constructor and Description |
---|
ZooKeeperCompletedCheckpointStore(int maxNumberOfCheckpointsToRetain,
ZooKeeperStateHandleStore<CompletedCheckpoint> checkpointsInZooKeeper,
Executor executor)
Creates a
ZooKeeperCompletedCheckpointStore instance. |
Modifier and Type | Method and Description |
---|---|
void |
addCheckpoint(CompletedCheckpoint checkpoint)
Synchronously writes the new checkpoints to ZooKeeper and asynchronously removes older ones.
|
static String |
checkpointIdToPath(long checkpointId)
Convert a checkpoint id into a ZooKeeper path.
|
List<CompletedCheckpoint> |
getAllCheckpoints()
Returns all
CompletedCheckpoint instances. |
int |
getMaxNumberOfRetainedCheckpoints()
Returns the max number of retained checkpoints.
|
int |
getNumberOfRetainedCheckpoints()
Returns the current number of retained checkpoints.
|
static long |
pathToCheckpointId(String path)
Converts a path to the checkpoint id.
|
void |
recover()
Gets the latest checkpoint from ZooKeeper and removes all others.
|
boolean |
requiresExternalizedCheckpoints()
This method returns whether the completed checkpoint store requires checkpoints to be
externalized.
|
void |
shutdown(JobStatus jobStatus)
Shuts down the store.
|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
getLatestCheckpoint, storeFor
public ZooKeeperCompletedCheckpointStore(int maxNumberOfCheckpointsToRetain, ZooKeeperStateHandleStore<CompletedCheckpoint> checkpointsInZooKeeper, Executor executor)
ZooKeeperCompletedCheckpointStore
instance.maxNumberOfCheckpointsToRetain
- The maximum number of checkpoints to retain (at least
1). Adding more checkpoints than this results in older checkpoints being discarded. On
recovery, we will only start with a single checkpoint.checkpointsInZooKeeper
- Completed checkpoints in ZooKeeperexecutor
- to execute blocking callspublic boolean requiresExternalizedCheckpoints()
CompletedCheckpointStore
requiresExternalizedCheckpoints
in interface CompletedCheckpointStore
public void recover() throws Exception
Important: Even if there are more than one checkpoint in ZooKeeper, this will only recover the latest and discard the others. Otherwise, there is no guarantee that the history of checkpoints is consistent.
recover
in interface CompletedCheckpointStore
Exception
public void addCheckpoint(CompletedCheckpoint checkpoint) throws Exception
addCheckpoint
in interface CompletedCheckpointStore
checkpoint
- Completed checkpoint to add.Exception
public List<CompletedCheckpoint> getAllCheckpoints() throws Exception
CompletedCheckpointStore
CompletedCheckpoint
instances.
Returns an empty list if no checkpoint has been added yet.
getAllCheckpoints
in interface CompletedCheckpointStore
Exception
public int getNumberOfRetainedCheckpoints()
CompletedCheckpointStore
getNumberOfRetainedCheckpoints
in interface CompletedCheckpointStore
public int getMaxNumberOfRetainedCheckpoints()
CompletedCheckpointStore
getMaxNumberOfRetainedCheckpoints
in interface CompletedCheckpointStore
public void shutdown(JobStatus jobStatus) throws Exception
CompletedCheckpointStore
The job status is forwarded and used to decide whether state should actually be discarded or kept.
shutdown
in interface CompletedCheckpointStore
jobStatus
- Job state on shut downException
public static String checkpointIdToPath(long checkpointId)
checkpointId
- to convert to the pathpublic static long pathToCheckpointId(String path)
path
- in ZooKeeperCopyright © 2014–2021 The Apache Software Foundation. All rights reserved.