org.apache.flink.runtime.highavailability

Interface HighAvailabilityServices

All Superinterfaces:

AutoCloseable

All Known Implementing Classes:

AbstractNonHaServices, AbstractYarnNonHaServices, EmbeddedHaServices, StandaloneHaServices, YarnHighAvailabilityServices, YarnIntraNonHaMasterServices, YarnPreConfiguredMasterNonHaServices, ZooKeeperHaServices

public interface HighAvailabilityServices
extends AutoCloseable

The HighAvailabilityServices give access to all services needed for a highly-available setup. In particular, the services provide access to highly available storage and registries, as well as distributed counters and leader election.

• ResourceManager leader election and leader retrieval
• JobManager leader election and leader retrieval
• Persistence for checkpoint metadata
• Registering the latest completed checkpoint(s)
• Persistence for the BLOB store
• Registry that marks a job's status
• Naming of RPC endpoints

Field Summary

Fields

Modifier and Type	Field and Description
static JobID	DEFAULT_JOB_ID This JobID should be used to identify the old JobManager when using the HighAvailabilityServices.
static UUID	DEFAULT_LEADER_ID This UUID should be used when no proper leader election happens, but a simple pre-configured leader is used.

Method Summary

Modifier and Type	Method and Description
void	close() Closes the high availability services, releasing all resources.
void	closeAndCleanupAllData() Closes the high availability services (releasing all resources) and deletes all data stored by these services in external stores.
BlobStore	createBlobStore() Creates the BLOB store in which BLOBs are stored in a highly-available fashion.
CheckpointRecoveryFactory	getCheckpointRecoveryFactory() Gets the checkpoint recovery factory for the job manager
LeaderElectionService	getDispatcherLeaderElectionService() Gets the leader election service for the cluster's dispatcher.
LeaderRetrievalService	getDispatcherLeaderRetriever() Gets the leader retriever for the dispatcher.
LeaderElectionService	getJobManagerLeaderElectionService(JobID jobID) Gets the leader election service for the given job.
LeaderRetrievalService	getJobManagerLeaderRetriever(JobID jobID) Deprecated. This method should only be used by the legacy code where the JobManager acts as the master.
LeaderRetrievalService	getJobManagerLeaderRetriever(JobID jobID, String defaultJobManagerAddress) Gets the leader retriever for the job JobMaster which is responsible for the given job
LeaderElectionService	getResourceManagerLeaderElectionService() Gets the leader election service for the cluster's resource manager.
LeaderRetrievalService	getResourceManagerLeaderRetriever() Gets the leader retriever for the cluster's resource manager.
RunningJobsRegistry	getRunningJobsRegistry() Gets the registry that holds information about whether jobs are currently running.
SubmittedJobGraphStore	getSubmittedJobGraphStore() Gets the submitted job graph store for the job manager
LeaderElectionService	getWebMonitorLeaderElectionService()
LeaderRetrievalService	getWebMonitorLeaderRetriever()

Field Detail

DEFAULT_LEADER_ID

static final UUID DEFAULT_LEADER_ID

This UUID should be used when no proper leader election happens, but a simple pre-configured leader is used. That is for example the case in non-highly-available standalone setups.

DEFAULT_JOB_ID

static final JobID DEFAULT_JOB_ID

This JobID should be used to identify the old JobManager when using the HighAvailabilityServices. With the new mode every JobMaster will have a distinct JobID assigned.

Method Detail

getResourceManagerLeaderRetriever

LeaderRetrievalService getResourceManagerLeaderRetriever()

Gets the leader retriever for the cluster's resource manager.

getDispatcherLeaderRetriever

LeaderRetrievalService getDispatcherLeaderRetriever()

Gets the leader retriever for the dispatcher. This leader retrieval service is not always accessible.

getJobManagerLeaderRetriever

@Deprecated
LeaderRetrievalService getJobManagerLeaderRetriever(JobID jobID)

Deprecated. This method should only be used by the legacy code where the JobManager acts as the master.

Gets the leader retriever for the job JobMaster which is responsible for the given job

Parameters:

jobID - The identifier of the job.

Returns:

Leader retrieval service to retrieve the job manager for the given job

getJobManagerLeaderRetriever

LeaderRetrievalService getJobManagerLeaderRetriever(JobID jobID,
                                                    String defaultJobManagerAddress)

Gets the leader retriever for the job JobMaster which is responsible for the given job

Parameters:

jobID - The identifier of the job.

defaultJobManagerAddress - JobManager address which will be returned by a static leader retrieval service.

Returns:

Leader retrieval service to retrieve the job manager for the given job

getWebMonitorLeaderRetriever

LeaderRetrievalService getWebMonitorLeaderRetriever()

getResourceManagerLeaderElectionService

LeaderElectionService getResourceManagerLeaderElectionService()

Gets the leader election service for the cluster's resource manager.

Returns:

Leader election service for the resource manager leader election

getDispatcherLeaderElectionService

LeaderElectionService getDispatcherLeaderElectionService()

Gets the leader election service for the cluster's dispatcher.

Returns:

Leader election service for the dispatcher leader election

getJobManagerLeaderElectionService

LeaderElectionService getJobManagerLeaderElectionService(JobID jobID)

Gets the leader election service for the given job.

Parameters:

jobID - The identifier of the job running the election.

Returns:

Leader election service for the job manager leader election

getWebMonitorLeaderElectionService

LeaderElectionService getWebMonitorLeaderElectionService()

getCheckpointRecoveryFactory

CheckpointRecoveryFactory getCheckpointRecoveryFactory()

Gets the checkpoint recovery factory for the job manager

Returns:

Checkpoint recovery factory

getSubmittedJobGraphStore

SubmittedJobGraphStore getSubmittedJobGraphStore()
                                          throws Exception

Gets the submitted job graph store for the job manager

Returns:

Submitted job graph store

Throws:

Exception - if the submitted job graph store could not be created

getRunningJobsRegistry

RunningJobsRegistry getRunningJobsRegistry()
                                    throws Exception

Gets the registry that holds information about whether jobs are currently running.

Returns:

Running job registry to retrieve running jobs

Throws:

Exception

createBlobStore

BlobStore createBlobStore()
                   throws IOException

Creates the BLOB store in which BLOBs are stored in a highly-available fashion.

Returns:

Blob store

Throws:

IOException - if the blob store could not be created

close

void close()
    throws Exception

Closes the high availability services, releasing all resources.

This method does not delete or clean up any data stored in external stores (file systems, ZooKeeper, etc). Another instance of the high availability services will be able to recover the job.

If an exception occurs during closing services, this method will attempt to continue closing other services and report exceptions only after all services have been attempted to be closed.

Specified by:

close in interface AutoCloseable

Throws:

Exception - Thrown, if an exception occurred while closing these services.

closeAndCleanupAllData

void closeAndCleanupAllData()
                     throws Exception

Closes the high availability services (releasing all resources) and deletes all data stored by these services in external stores.

After this method was called, the any job or session that was managed by these high availability services will be unrecoverable.

If an exception occurs during cleanup, this method will attempt to continue the cleanup and report exceptions only after all cleanup steps have been attempted.

Throws:

Exception - Thrown, if an exception occurred while closing these services or cleaning up data stored by them.