public class ZooKeeperCompletedCheckpointStore extends Object implements CompletedCheckpointStore
CompletedCheckpointStore
for JobManagers running in RecoveryMode.ZOOKEEPER
.
Checkpoints are added under a ZNode per job:
+----O /flink/checkpoints/<job-id> [persistent] . | . +----O /flink/checkpoints/<job-id>/1 [persistent] . . . . . . . . . . +----O /flink/checkpoints/<job-id>/N [persistent]
During recovery, the latest checkpoint is read from ZooKeeper. If there is more than one, only the latest one is used and older ones are discarded (even if the maximum number of retained checkpoints is greater than one).
If there is a network partition and multiple JobManagers run concurrent checkpoints for the same program, it is OK to take any valid successful checkpoint as long as the "history" of checkpoints is consistent. Currently, after recovery we start out with only a single checkpoint to circumvent those situations.
Constructor and Description |
---|
ZooKeeperCompletedCheckpointStore(int maxNumberOfCheckpointsToRetain,
ClassLoader userClassLoader,
org.apache.curator.framework.CuratorFramework client,
String checkpointsPath,
StateStorageHelper<CompletedCheckpoint> stateStorage)
Creates a
ZooKeeperCompletedCheckpointStore instance. |
Modifier and Type | Method and Description |
---|---|
void |
addCheckpoint(CompletedCheckpoint checkpoint)
Synchronously writes the new checkpoints to ZooKeeper and asynchronously removes older ones.
|
void |
discardAllCheckpoints()
Discards all added
CompletedCheckpoint instances via CompletedCheckpoint.discard(ClassLoader) . |
List<CompletedCheckpoint> |
getAllCheckpoints()
Returns all
CompletedCheckpoint instances. |
CompletedCheckpoint |
getLatestCheckpoint()
Returns the latest
CompletedCheckpoint instance or null if none was
added. |
int |
getNumberOfRetainedCheckpoints()
Returns the current number of retained checkpoints.
|
void |
recover()
Gets the latest checkpoint from ZooKeeper and removes all others.
|
public ZooKeeperCompletedCheckpointStore(int maxNumberOfCheckpointsToRetain, ClassLoader userClassLoader, org.apache.curator.framework.CuratorFramework client, String checkpointsPath, StateStorageHelper<CompletedCheckpoint> stateStorage) throws Exception
ZooKeeperCompletedCheckpointStore
instance.maxNumberOfCheckpointsToRetain
- The maximum number of checkpoints to retain (at
least 1). Adding more checkpoints than this results
in older checkpoints being discarded. On recovery,
we will only start with a single checkpoint.userClassLoader
- The user class loader used to discard checkpointsclient
- The Curator ZooKeeper clientcheckpointsPath
- The ZooKeeper path for the checkpoints (needs to
start with a '/')stateStorage
- State storage to be used to persist the completed
checkpointException
public void recover() throws Exception
Important: Even if there are more than one checkpoint in ZooKeeper, this will only recover the latest and discard the others. Otherwise, there is no guarantee that the history of checkpoints is consistent.
recover
in interface CompletedCheckpointStore
Exception
public void addCheckpoint(CompletedCheckpoint checkpoint) throws Exception
addCheckpoint
in interface CompletedCheckpointStore
checkpoint
- Completed checkpoint to add.Exception
public CompletedCheckpoint getLatestCheckpoint() throws Exception
CompletedCheckpointStore
CompletedCheckpoint
instance or null
if none was
added.getLatestCheckpoint
in interface CompletedCheckpointStore
Exception
public List<CompletedCheckpoint> getAllCheckpoints() throws Exception
CompletedCheckpointStore
CompletedCheckpoint
instances.
Returns an empty list if no checkpoint has been added yet.
getAllCheckpoints
in interface CompletedCheckpointStore
Exception
public int getNumberOfRetainedCheckpoints()
CompletedCheckpointStore
getNumberOfRetainedCheckpoints
in interface CompletedCheckpointStore
public void discardAllCheckpoints() throws Exception
CompletedCheckpointStore
CompletedCheckpoint
instances via CompletedCheckpoint.discard(ClassLoader)
.discardAllCheckpoints
in interface CompletedCheckpointStore
Exception
Copyright © 2014–2017 The Apache Software Foundation. All rights reserved.