org.apache.flink.runtime.checkpoint

Interface MasterTriggerRestoreHook<T>

Type Parameters:

T - The type of the data produced by the hook and stored as part of the checkpoint metadata. If the hook never stores any data, this can be typed to Void.

public interface MasterTriggerRestoreHook<T>

The interface for hooks that can be called by the checkpoint coordinator when triggering or restoring a checkpoint. Such a hook is useful for example when preparing external systems for taking or restoring checkpoints.

The triggerCheckpoint(long, long, Executor) method (called when triggering a checkpoint) can return a result (via a future) that will be stored as part of the checkpoint metadata. When restoring a checkpoint, that stored result will be given to the restoreCheckpoint(long, Object) method. The hook's identifier is used to map data to hook in the presence of multiple hooks, and when resuming a savepoint that was potentially created by a different job. The identifier has a similar role as for example the operator UID in the streaming API.

It is possible that a job fails (and is subsequently restarted) before any checkpoints were successful. In that situation, the checkpoint coordination calls reset() to give the hook an opportunity to, for example, reset an external system to initial conditions.

The MasterTriggerRestoreHook is defined when creating the streaming dataflow graph. It is attached to the job graph, which gets sent to the cluster for execution. To avoid having to make the hook itself serializable, these hooks are attached to the job graph via a MasterTriggerRestoreHook.Factory.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static interface	MasterTriggerRestoreHook.Factory A factory to instantiate a MasterTriggerRestoreHook.

Method Summary

Modifier and Type	Method and Description
default void	close() Tear-down method for the hook.
SimpleVersionedSerializer<T>	createCheckpointDataSerializer() Creates a the serializer to (de)serializes the data stored by this hook.
String	getIdentifier() Gets the identifier of this hook.
default void	reset() This method is called by the checkpoint coordinator to reset the hook when execution is restarted in the absence of any checkpoint state.
void	restoreCheckpoint(long checkpointId, T checkpointData) This method is called by the checkpoint coordinator prior to restoring the state of a checkpoint.
CompletableFuture<T>	triggerCheckpoint(long checkpointId, long timestamp, Executor executor) This method is called by the checkpoint coordinator prior when triggering a checkpoint, prior to sending the "trigger checkpoint" messages to the source tasks.

Method Detail

getIdentifier

String getIdentifier()

Gets the identifier of this hook. The identifier is used to identify a specific hook in the presence of multiple hooks and to give it the correct checkpointed data upon checkpoint restoration.

The identifier should be unique between different hooks of a job, but deterministic/constant so that upon resuming a savepoint, the hook will get the correct data. For example, if the hook calls into another storage system and persists namespace/schema specific information, then the name of the storage system, together with the namespace/schema name could be an appropriate identifier.

When multiple hooks of the same name are created and attached to a job graph, only the first one is actually used. This can be exploited to deduplicate hooks that would do the same thing.

Returns:

The identifier of the hook.

reset

default void reset()
            throws Exception

This method is called by the checkpoint coordinator to reset the hook when execution is restarted in the absence of any checkpoint state.

Throws:

Exception - Exceptions encountered when calling the hook will cause execution to fail.

close

default void close()
            throws Exception

Tear-down method for the hook.

Throws:

Exception - Exceptions encountered when calling close will be logged.

triggerCheckpoint

@Nullable
CompletableFuture<T> triggerCheckpoint(long checkpointId,
                                                 long timestamp,
                                                 Executor executor)
                                          throws Exception

This method is called by the checkpoint coordinator prior when triggering a checkpoint, prior to sending the "trigger checkpoint" messages to the source tasks.

If the hook implementation wants to store data as part of the checkpoint, it may return that data via a future, otherwise it should return null. The data is stored as part of the checkpoint metadata under the hooks identifier (see getIdentifier()).

If the action by this hook needs to be executed synchronously, then this method should directly execute the action synchronously and block until it is complete. The returned future (if any) would typically be a completed future.

If the action should be executed asynchronously and only needs to complete before the checkpoint is considered completed, then the method may use the given executor to execute the actual action and would signal its completion by completing the future. For hooks that do not need to store data, the future would be completed with null.

Parameters:

checkpointId - The ID (logical timestamp, monotonously increasing) of the checkpoint

timestamp - The wall clock timestamp when the checkpoint was triggered, for info/logging purposes.

executor - The executor for asynchronous actions

Returns:

Optionally, a future that signals when the hook has completed and that contains data to be stored with the checkpoint.

Throws:

Exception - Exceptions encountered when calling the hook will cause the checkpoint to abort.

restoreCheckpoint

void restoreCheckpoint(long checkpointId,
                       @Nullable
                       T checkpointData)
                throws Exception

This method is called by the checkpoint coordinator prior to restoring the state of a checkpoint. If the checkpoint did store data from this hook, that data will be passed to this method.

Parameters:

checkpointId - The ID (logical timestamp) of the restored checkpoint

checkpointData - The data originally stored in the checkpoint by this hook, possibly null.

Throws:

Exception - Exceptions thrown while restoring the checkpoint will cause the restore operation to fail and to possibly fall back to another checkpoint.

createCheckpointDataSerializer

@Nullable
SimpleVersionedSerializer<T> createCheckpointDataSerializer()

Creates a the serializer to (de)serializes the data stored by this hook. The serializer serializes the result of the Future returned by the triggerCheckpoint(long, long, Executor) method, and deserializes the data stored in the checkpoint into the object passed to the restoreCheckpoint(long, Object) method.

If the hook never returns any data to be stored, then this method may return null as the serializer.

Returns:

The serializer to (de)serializes the data stored by this hook