T- The type of the data produced by the hook and stored as part of the checkpoint metadata. If the hook never stores any data, this can be typed to
public interface MasterTriggerRestoreHook<T>
triggerCheckpoint(long, long, Executor) method (called when triggering a
checkpoint) can return a result (via a future) that will be stored as part of the checkpoint
metadata. When restoring a checkpoint, that stored result will be given to the
restoreCheckpoint(long, Object) method. The hook's
identifier is used
to map data to hook in the presence of multiple hooks, and when resuming a savepoint that was
potentially created by a different job. The identifier has a similar role as for example the
operator UID in the streaming API.
It is possible that a job fails (and is subsequently restarted) before any checkpoints were
successful. In that situation, the checkpoint coordination calls
reset() to give the
hook an opportunity to, for example, reset an external system to initial conditions.
The MasterTriggerRestoreHook is defined when creating the streaming dataflow graph. It is
attached to the job graph, which gets sent to the cluster for execution. To avoid having to make
the hook itself serializable, these hooks are attached to the job graph via a
|Modifier and Type||Interface and Description|
A factory to instantiate a
|Modifier and Type||Method and Description|
Tear-down method for the hook.
Creates a the serializer to (de)serializes the data stored by this hook.
Gets the identifier of this hook.
This method is called by the checkpoint coordinator to reset the hook when execution is restarted in the absence of any checkpoint state.
This method is called by the checkpoint coordinator prior to restoring the state of a checkpoint.
This method is called by the checkpoint coordinator prior when triggering a checkpoint, prior to sending the "trigger checkpoint" messages to the source tasks.
The identifier should be unique between different hooks of a job, but deterministic/constant so that upon resuming a savepoint, the hook will get the correct data. For example, if the hook calls into another storage system and persists namespace/schema specific information, then the name of the storage system, together with the namespace/schema name could be an appropriate identifier.
When multiple hooks of the same name are created and attached to a job graph, only the first one is actually used. This can be exploited to deduplicate hooks that would do the same thing.
default void reset() throws Exception
Exception- Exceptions encountered when calling the hook will cause execution to fail.
default void close() throws Exception
Exception- Exceptions encountered when calling close will be logged.
@Nullable CompletableFuture<T> triggerCheckpoint(long checkpointId, long timestamp, Executor executor) throws Exception
If the hook implementation wants to store data as part of the checkpoint, it may return
that data via a future, otherwise it should return null. The data is stored as part of the
checkpoint metadata under the hooks identifier (see
If the action by this hook needs to be executed synchronously, then this method should directly execute the action synchronously. The returned future (if any) would typically be a completed future.
If the action should be executed asynchronously and only needs to complete before the checkpoint is considered completed, then the method may use the given executor to execute the actual action and would signal its completion by completing the future. For hooks that do not need to store data, the future would be completed with null.
Please note that this method should be non-blocking. Any heavy operation like IO operation should be executed asynchronously with given executor.
checkpointId- The ID (logical timestamp, monotonously increasing) of the checkpoint
timestamp- The wall clock timestamp when the checkpoint was triggered, for info/logging purposes.
executor- The executor for asynchronous actions
Exception- Exceptions encountered when calling the hook will cause the checkpoint to abort.
void restoreCheckpoint(long checkpointId, @Nullable T checkpointData) throws Exception
checkpointId- The ID (logical timestamp) of the restored checkpoint
checkpointData- The data originally stored in the checkpoint by this hook, possibly null.
Exception- Exceptions thrown while restoring the checkpoint will cause the restore operation to fail and to possibly fall back to another checkpoint.
@Nullable SimpleVersionedSerializer<T> createCheckpointDataSerializer()
triggerCheckpoint(long, long, Executor)method, and deserializes the data stored in the checkpoint into the object passed to the
If the hook never returns any data to be stored, then this method may return null as the serializer.
Copyright © 2014–2023 The Apache Software Foundation. All rights reserved.