org.apache.flink.runtime.highavailability

Interface HighAvailabilityServices

All Superinterfaces:

AutoCloseable, ClientHighAvailabilityServices, GloballyCleanableResource

All Known Implementing Classes:

AbstractHaServices, AbstractNonHaServices, EmbeddedHaServices, EmbeddedHaServicesWithLeadershipControl, KubernetesLeaderElectionHaServices, StandaloneHaServices, ZooKeeperLeaderElectionHaServices

public interface HighAvailabilityServices
extends ClientHighAvailabilityServices, GloballyCleanableResource

The HighAvailabilityServices give access to all services needed for a highly-available setup. In particular, the services provide access to highly available storage and registries, as well as distributed counters and leader election.

• ResourceManager leader election and leader retrieval
• JobManager leader election and leader retrieval
• Persistence for checkpoint metadata
• Registering the latest completed checkpoint(s)
• Persistence for the BLOB store
• Registry that marks a job's status
• Naming of RPC endpoints

Field Summary

Fields

Modifier and Type	Field and Description
static JobID	DEFAULT_JOB_ID This JobID should be used to identify the old JobManager when using the HighAvailabilityServices.
static UUID	DEFAULT_LEADER_ID This UUID should be used when no proper leader election happens, but a simple pre-configured leader is used.

Method Summary

Modifier and Type	Method and Description
void	cleanupAllData() Deletes all data stored by high availability services in external stores.
void	close() Closes the high availability services, releasing all resources.
default void	closeWithOptionalClean(boolean cleanupData) Calls cleanupAllData() (if true is passed as a parameter) before calling close() on this instance.
BlobStore	createBlobStore() Creates the BLOB store in which BLOBs are stored in a highly-available fashion.
CheckpointRecoveryFactory	getCheckpointRecoveryFactory() Gets the checkpoint recovery factory for the job manager.
default LeaderElection	getClusterRestEndpointLeaderElection() Gets the LeaderElection for the cluster's rest endpoint.
default LeaderRetrievalService	getClusterRestEndpointLeaderRetriever() Get the leader retriever for the cluster's rest endpoint.
LeaderElection	getDispatcherLeaderElection() Gets the LeaderElection for the cluster's dispatcher.
LeaderRetrievalService	getDispatcherLeaderRetriever() Gets the leader retriever for the dispatcher.
JobGraphStore	getJobGraphStore() Gets the submitted job graph store for the job manager.
LeaderElection	getJobManagerLeaderElection(JobID jobID) Gets the LeaderElection for the job with the given JobID.
LeaderRetrievalService	getJobManagerLeaderRetriever(JobID jobID) Deprecated. This method should only be used by the legacy code where the JobManager acts as the master.
LeaderRetrievalService	getJobManagerLeaderRetriever(JobID jobID, String defaultJobManagerAddress) Gets the leader retriever for the job JobMaster which is responsible for the given job.
JobResultStore	getJobResultStore() Gets the store that holds information about the state of finished jobs.
LeaderElection	getResourceManagerLeaderElection() Gets the LeaderElection for the cluster's resource manager.
LeaderRetrievalService	getResourceManagerLeaderRetriever() Gets the leader retriever for the cluster's resource manager.
default LeaderElection	getWebMonitorLeaderElection() Deprecated. Use getClusterRestEndpointLeaderElection() instead.
default LeaderRetrievalService	getWebMonitorLeaderRetriever() Deprecated. just use getClusterRestEndpointLeaderRetriever() instead of this method.
default CompletableFuture<Void>	globalCleanupAsync(JobID jobId, Executor executor) globalCleanupAsync is expected to be called from the main thread.

Field Detail

DEFAULT_LEADER_ID

static final UUID DEFAULT_LEADER_ID

This UUID should be used when no proper leader election happens, but a simple pre-configured leader is used. That is for example the case in non-highly-available standalone setups.

DEFAULT_JOB_ID

static final JobID DEFAULT_JOB_ID

This JobID should be used to identify the old JobManager when using the HighAvailabilityServices. With the new mode every JobMaster will have a distinct JobID assigned.

Method Detail

getResourceManagerLeaderRetriever

LeaderRetrievalService getResourceManagerLeaderRetriever()

Gets the leader retriever for the cluster's resource manager.

getDispatcherLeaderRetriever

LeaderRetrievalService getDispatcherLeaderRetriever()

Gets the leader retriever for the dispatcher. This leader retrieval service is not always accessible.

getJobManagerLeaderRetriever

@Deprecated
LeaderRetrievalService getJobManagerLeaderRetriever(JobID jobID)

Deprecated. This method should only be used by the legacy code where the JobManager acts as the master.

Gets the leader retriever for the job JobMaster which is responsible for the given job.

Parameters:

jobID - The identifier of the job.

Returns:

Leader retrieval service to retrieve the job manager for the given job

getJobManagerLeaderRetriever

LeaderRetrievalService getJobManagerLeaderRetriever(JobID jobID,
                                                    String defaultJobManagerAddress)

Gets the leader retriever for the job JobMaster which is responsible for the given job.

Parameters:

jobID - The identifier of the job.

defaultJobManagerAddress - JobManager address which will be returned by a static leader retrieval service.

Returns:

Leader retrieval service to retrieve the job manager for the given job

getWebMonitorLeaderRetriever

@Deprecated
default LeaderRetrievalService getWebMonitorLeaderRetriever()

Deprecated. just use getClusterRestEndpointLeaderRetriever() instead of this method.

This retriever should no longer be used on the cluster side. The web monitor retriever is only required on the client-side and we have a dedicated high-availability services for the client, named ClientHighAvailabilityServices. See also FLINK-13750.

Returns:

the leader retriever for web monitor

getResourceManagerLeaderElection

LeaderElection getResourceManagerLeaderElection()

Gets the LeaderElection for the cluster's resource manager.

getDispatcherLeaderElection

LeaderElection getDispatcherLeaderElection()

Gets the LeaderElection for the cluster's dispatcher.

getJobManagerLeaderElection

LeaderElection getJobManagerLeaderElection(JobID jobID)

Gets the LeaderElection for the job with the given JobID.

getWebMonitorLeaderElection

@Deprecated
default LeaderElection getWebMonitorLeaderElection()

Deprecated. Use getClusterRestEndpointLeaderElection() instead.

Gets the LeaderElection for the cluster's rest endpoint.

getCheckpointRecoveryFactory

CheckpointRecoveryFactory getCheckpointRecoveryFactory()
                                                throws Exception

Gets the checkpoint recovery factory for the job manager.

Returns:

Checkpoint recovery factory

Throws:

Exception

getJobGraphStore

JobGraphStore getJobGraphStore()
                        throws Exception

Gets the submitted job graph store for the job manager.

Returns:

Submitted job graph store

Throws:

Exception - if the submitted job graph store could not be created

getJobResultStore

JobResultStore getJobResultStore()
                          throws Exception

Gets the store that holds information about the state of finished jobs.

Returns:

Store of finished job results

Throws:

Exception - if job result store could not be created

createBlobStore

BlobStore createBlobStore()
                   throws IOException

Creates the BLOB store in which BLOBs are stored in a highly-available fashion.

Returns:

Blob store

Throws:

IOException - if the blob store could not be created

getClusterRestEndpointLeaderElection

default LeaderElection getClusterRestEndpointLeaderElection()

Gets the LeaderElection for the cluster's rest endpoint.

getClusterRestEndpointLeaderRetriever

default LeaderRetrievalService getClusterRestEndpointLeaderRetriever()

Description copied from interface: ClientHighAvailabilityServices

Get the leader retriever for the cluster's rest endpoint.

Specified by:

getClusterRestEndpointLeaderRetriever in interface ClientHighAvailabilityServices

Returns:

the leader retriever for cluster's rest endpoint.

close

void close()
    throws Exception

Closes the high availability services, releasing all resources.

This method does not delete or clean up any data stored in external stores (file systems, ZooKeeper, etc). Another instance of the high availability services will be able to recover the job.

If an exception occurs during closing services, this method will attempt to continue closing other services and report exceptions only after all services have been attempted to be closed.

Specified by:

close in interface AutoCloseable

Throws:

Exception - Thrown, if an exception occurred while closing these services.

cleanupAllData

void cleanupAllData()
             throws Exception

Deletes all data stored by high availability services in external stores.

After this method was called, any job or session that was managed by these high availability services will be unrecoverable.

If an exception occurs during cleanup, this method will attempt to continue the cleanup and report exceptions only after all cleanup steps have been attempted.

Throws:

Exception - if an error occurred while cleaning up data stored by them.

closeWithOptionalClean

default void closeWithOptionalClean(boolean cleanupData)
                             throws Exception

Calls cleanupAllData() (if true is passed as a parameter) before calling close() on this instance. Any error that appeared during the cleanup will be propagated after calling close().

Throws:

Exception

globalCleanupAsync

default CompletableFuture<Void> globalCleanupAsync(JobID jobId,
                                                   Executor executor)

Description copied from interface: GloballyCleanableResource

globalCleanupAsync is expected to be called from the main thread. Heavy IO tasks should be outsourced into the passed cleanupExecutor. Thread-safety must be ensured.

Specified by:

globalCleanupAsync in interface GloballyCleanableResource

Parameters:

jobId - The JobID of the job for which the local data should be cleaned up.

executor - The fallback executor for IO-heavy operations.

Returns:

The cleanup result future.