org.apache.flink.runtime.highavailability

Interface HighAvailabilityServices

All Superinterfaces:

AutoCloseable, ClientHighAvailabilityServices, GloballyCleanableResource

All Known Implementing Classes:

AbstractHaServices, AbstractNonHaServices, AbstractZooKeeperHaServices, EmbeddedHaServices, EmbeddedHaServicesWithLeadershipControl, KubernetesMultipleComponentLeaderElectionHaServices, StandaloneHaServices, ZooKeeperMultipleComponentLeaderElectionHaServices

public interface HighAvailabilityServices
extends ClientHighAvailabilityServices, GloballyCleanableResource

The HighAvailabilityServices give access to all services needed for a highly-available setup. In particular, the services provide access to highly available storage and registries, as well as distributed counters and leader election.

• ResourceManager leader election and leader retrieval
• JobManager leader election and leader retrieval
• Persistence for checkpoint metadata
• Registering the latest completed checkpoint(s)
• Persistence for the BLOB store
• Registry that marks a job's status
• Naming of RPC endpoints

Field Summary

Fields

Modifier and Type	Field and Description
static JobID	DEFAULT_JOB_ID This JobID should be used to identify the old JobManager when using the HighAvailabilityServices.
static UUID	DEFAULT_LEADER_ID This UUID should be used when no proper leader election happens, but a simple pre-configured leader is used.

Method Summary

Modifier and Type	Method and Description
void	close() Closes the high availability services, releasing all resources.
void	closeAndCleanupAllData() Closes the high availability services (releasing all resources) and deletes all data stored by these services in external stores.
BlobStore	createBlobStore() Creates the BLOB store in which BLOBs are stored in a highly-available fashion.
CheckpointRecoveryFactory	getCheckpointRecoveryFactory() Gets the checkpoint recovery factory for the job manager.
default LeaderElectionService	getClusterRestEndpointLeaderElectionService() Gets the leader election service for the cluster's rest endpoint.
default LeaderRetrievalService	getClusterRestEndpointLeaderRetriever() Get the leader retriever for the cluster's rest endpoint.
LeaderElectionService	getDispatcherLeaderElectionService() Gets the leader election service for the cluster's dispatcher.
LeaderRetrievalService	getDispatcherLeaderRetriever() Gets the leader retriever for the dispatcher.
JobGraphStore	getJobGraphStore() Gets the submitted job graph store for the job manager.
LeaderElectionService	getJobManagerLeaderElectionService(JobID jobID) Gets the leader election service for the given job.
LeaderRetrievalService	getJobManagerLeaderRetriever(JobID jobID) Deprecated. This method should only be used by the legacy code where the JobManager acts as the master.
LeaderRetrievalService	getJobManagerLeaderRetriever(JobID jobID, String defaultJobManagerAddress) Gets the leader retriever for the job JobMaster which is responsible for the given job.
JobResultStore	getJobResultStore() Gets the store that holds information about the state of finished jobs.
LeaderElectionService	getResourceManagerLeaderElectionService() Gets the leader election service for the cluster's resource manager.
LeaderRetrievalService	getResourceManagerLeaderRetriever() Gets the leader retriever for the cluster's resource manager.
default LeaderElectionService	getWebMonitorLeaderElectionService() Deprecated. Use getClusterRestEndpointLeaderElectionService() instead.
default LeaderRetrievalService	getWebMonitorLeaderRetriever() Deprecated. just use getClusterRestEndpointLeaderRetriever() instead of this method.
default CompletableFuture<Void>	globalCleanupAsync(JobID jobId, Executor executor) globalCleanupAsync is expected to be called from the main thread.

Field Detail

DEFAULT_LEADER_ID

static final UUID DEFAULT_LEADER_ID

This UUID should be used when no proper leader election happens, but a simple pre-configured leader is used. That is for example the case in non-highly-available standalone setups.

DEFAULT_JOB_ID

static final JobID DEFAULT_JOB_ID

This JobID should be used to identify the old JobManager when using the HighAvailabilityServices. With the new mode every JobMaster will have a distinct JobID assigned.

Method Detail

getResourceManagerLeaderRetriever

LeaderRetrievalService getResourceManagerLeaderRetriever()

Gets the leader retriever for the cluster's resource manager.

getDispatcherLeaderRetriever

LeaderRetrievalService getDispatcherLeaderRetriever()

Gets the leader retriever for the dispatcher. This leader retrieval service is not always accessible.

getJobManagerLeaderRetriever

@Deprecated
LeaderRetrievalService getJobManagerLeaderRetriever(JobID jobID)

Deprecated. This method should only be used by the legacy code where the JobManager acts as the master.

Gets the leader retriever for the job JobMaster which is responsible for the given job.

Parameters:

jobID - The identifier of the job.

Returns:

Leader retrieval service to retrieve the job manager for the given job

getJobManagerLeaderRetriever

LeaderRetrievalService getJobManagerLeaderRetriever(JobID jobID,
                                                    String defaultJobManagerAddress)

Gets the leader retriever for the job JobMaster which is responsible for the given job.

Parameters:

jobID - The identifier of the job.

defaultJobManagerAddress - JobManager address which will be returned by a static leader retrieval service.

Returns:

Leader retrieval service to retrieve the job manager for the given job

getWebMonitorLeaderRetriever

@Deprecated
default LeaderRetrievalService getWebMonitorLeaderRetriever()

Deprecated. just use getClusterRestEndpointLeaderRetriever() instead of this method.

This retriever should no longer be used on the cluster side. The web monitor retriever is only required on the client-side and we have a dedicated high-availability services for the client, named ClientHighAvailabilityServices. See also FLINK-13750.

Returns:

the leader retriever for web monitor

getResourceManagerLeaderElectionService

LeaderElectionService getResourceManagerLeaderElectionService()

Gets the leader election service for the cluster's resource manager.

Returns:

Leader election service for the resource manager leader election

getDispatcherLeaderElectionService

LeaderElectionService getDispatcherLeaderElectionService()

Gets the leader election service for the cluster's dispatcher.

Returns:

Leader election service for the dispatcher leader election

getJobManagerLeaderElectionService

LeaderElectionService getJobManagerLeaderElectionService(JobID jobID)

Gets the leader election service for the given job.

Parameters:

jobID - The identifier of the job running the election.

Returns:

Leader election service for the job manager leader election

getWebMonitorLeaderElectionService

@Deprecated
default LeaderElectionService getWebMonitorLeaderElectionService()

Deprecated. Use getClusterRestEndpointLeaderElectionService() instead.

Gets the leader election service for the cluster's rest endpoint.

Returns:

the leader election service used by the cluster's rest endpoint

getCheckpointRecoveryFactory

CheckpointRecoveryFactory getCheckpointRecoveryFactory()
                                                throws Exception

Gets the checkpoint recovery factory for the job manager.

Returns:

Checkpoint recovery factory

Throws:

Exception

getJobGraphStore

JobGraphStore getJobGraphStore()
                        throws Exception

Gets the submitted job graph store for the job manager.

Returns:

Submitted job graph store

Throws:

Exception - if the submitted job graph store could not be created

getJobResultStore

JobResultStore getJobResultStore()
                          throws Exception

Gets the store that holds information about the state of finished jobs.

Returns:

Store of finished job results

Throws:

Exception - if job result store could not be created

createBlobStore

BlobStore createBlobStore()
                   throws IOException

Creates the BLOB store in which BLOBs are stored in a highly-available fashion.

Returns:

Blob store

Throws:

IOException - if the blob store could not be created

getClusterRestEndpointLeaderElectionService

default LeaderElectionService getClusterRestEndpointLeaderElectionService()

Gets the leader election service for the cluster's rest endpoint.

Returns:

the leader election service used by the cluster's rest endpoint

getClusterRestEndpointLeaderRetriever

default LeaderRetrievalService getClusterRestEndpointLeaderRetriever()

Description copied from interface: ClientHighAvailabilityServices

Get the leader retriever for the cluster's rest endpoint.

Specified by:

getClusterRestEndpointLeaderRetriever in interface ClientHighAvailabilityServices

Returns:

the leader retriever for cluster's rest endpoint.

close

void close()
    throws Exception

Closes the high availability services, releasing all resources.

This method does not delete or clean up any data stored in external stores (file systems, ZooKeeper, etc). Another instance of the high availability services will be able to recover the job.

If an exception occurs during closing services, this method will attempt to continue closing other services and report exceptions only after all services have been attempted to be closed.

Specified by:

close in interface AutoCloseable

Throws:

Exception - Thrown, if an exception occurred while closing these services.

closeAndCleanupAllData

void closeAndCleanupAllData()
                     throws Exception

Closes the high availability services (releasing all resources) and deletes all data stored by these services in external stores.

After this method was called, the any job or session that was managed by these high availability services will be unrecoverable.

If an exception occurs during cleanup, this method will attempt to continue the cleanup and report exceptions only after all cleanup steps have been attempted.

Throws:

Exception - Thrown, if an exception occurred while closing these services or cleaning up data stored by them.

globalCleanupAsync

default CompletableFuture<Void> globalCleanupAsync(JobID jobId,
                                                   Executor executor)

Description copied from interface: GloballyCleanableResource

globalCleanupAsync is expected to be called from the main thread. Heavy IO tasks should be outsourced into the passed cleanupExecutor. Thread-safety must be ensured.

Specified by:

globalCleanupAsync in interface GloballyCleanableResource

Parameters:

jobId - The JobID of the job for which the local data should be cleaned up.

executor - The fallback executor for IO-heavy operations.

Returns:

The cleanup result future.