org.apache.flink.runtime.highavailability.zookeeper

Class ZooKeeperHaServices

java.lang.Object

org.apache.flink.runtime.highavailability.zookeeper.ZooKeeperHaServices

All Implemented Interfaces:

AutoCloseable, ClientHighAvailabilityServices, HighAvailabilityServices

public class ZooKeeperHaServices
extends Object
implements HighAvailabilityServices

An implementation of the HighAvailabilityServices using Apache ZooKeeper. The services store data in ZooKeeper's nodes as illustrated by the following tree structure:

 /flink
      +/cluster_id_1/resource_manager_lock
      |            |
      |            +/job-id-1/job_manager_lock
      |            |         /checkpoints/latest
      |            |                     /latest-1
      |            |                     /latest-2
      |            |
      |            +/job-id-2/job_manager_lock
      |
      +/cluster_id_2/resource_manager_lock
                   |
                   +/job-id-1/job_manager_lock
                            |/checkpoints/latest
                            |            /latest-1
                            |/persisted_job_graph
 

The root path "/flink" is configurable via the option HighAvailabilityOptions.HA_ZOOKEEPER_ROOT. This makes sure Flink stores its data under specific subtrees in ZooKeeper, for example to accommodate specific permission.

The "cluster_id" part identifies the data stored for a specific Flink "cluster". This "cluster" can be either a standalone or containerized Flink cluster, or it can be job on a framework like YARN or Mesos (in a "per-job-cluster" mode).

In case of a "per-job-cluster" on YARN or Mesos, the cluster-id is generated and configured automatically by the client or dispatcher that submits the Job to YARN or Mesos.

In the case of a standalone cluster, that cluster-id needs to be configured via HighAvailabilityOptions.HA_CLUSTER_ID. All nodes with the same cluster id will join the same cluster and participate in the execution of the same set of jobs.

Field Summary

Fields inherited from interface org.apache.flink.runtime.highavailability.HighAvailabilityServices

DEFAULT_JOB_ID, DEFAULT_LEADER_ID

Constructor Summary

Constructors

Constructor and Description
ZooKeeperHaServices(org.apache.curator.framework.CuratorFramework client, Executor executor, Configuration configuration, BlobStoreService blobStoreService)

Method Summary

Modifier and Type	Method and Description
void	close() Closes the high availability services, releasing all resources.
void	closeAndCleanupAllData() Closes the high availability services (releasing all resources) and deletes all data stored by these services in external stores.
BlobStore	createBlobStore() Creates the BLOB store in which BLOBs are stored in a highly-available fashion.
CheckpointRecoveryFactory	getCheckpointRecoveryFactory() Gets the checkpoint recovery factory for the job manager.
LeaderElectionService	getClusterRestEndpointLeaderElectionService() Gets the leader election service for the cluster's rest endpoint.
LeaderRetrievalService	getClusterRestEndpointLeaderRetriever() Get the leader retriever for the cluster's rest endpoint.
LeaderElectionService	getDispatcherLeaderElectionService() Gets the leader election service for the cluster's dispatcher.
LeaderRetrievalService	getDispatcherLeaderRetriever() Gets the leader retriever for the dispatcher.
JobGraphStore	getJobGraphStore() Gets the submitted job graph store for the job manager.
LeaderElectionService	getJobManagerLeaderElectionService(JobID jobID) Gets the leader election service for the given job.
LeaderRetrievalService	getJobManagerLeaderRetriever(JobID jobID) Gets the leader retriever for the job JobMaster which is responsible for the given job.
LeaderRetrievalService	getJobManagerLeaderRetriever(JobID jobID, String defaultJobManagerAddress) Gets the leader retriever for the job JobMaster which is responsible for the given job.
LeaderElectionService	getResourceManagerLeaderElectionService() Gets the leader election service for the cluster's resource manager.
LeaderRetrievalService	getResourceManagerLeaderRetriever() Gets the leader retriever for the cluster's resource manager.
RunningJobsRegistry	getRunningJobsRegistry() Gets the registry that holds information about whether jobs are currently running.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.highavailability.HighAvailabilityServices

getWebMonitorLeaderElectionService, getWebMonitorLeaderRetriever

Constructor Detail

ZooKeeperHaServices

public ZooKeeperHaServices(org.apache.curator.framework.CuratorFramework client,
                           Executor executor,
                           Configuration configuration,
                           BlobStoreService blobStoreService)

Method Detail

getResourceManagerLeaderRetriever

public LeaderRetrievalService getResourceManagerLeaderRetriever()

Description copied from interface: HighAvailabilityServices

Gets the leader retriever for the cluster's resource manager.

Specified by:

getResourceManagerLeaderRetriever in interface HighAvailabilityServices

getDispatcherLeaderRetriever

public LeaderRetrievalService getDispatcherLeaderRetriever()

Description copied from interface: HighAvailabilityServices

Gets the leader retriever for the dispatcher. This leader retrieval service is not always accessible.

Specified by:

getDispatcherLeaderRetriever in interface HighAvailabilityServices

getJobManagerLeaderRetriever

public LeaderRetrievalService getJobManagerLeaderRetriever(JobID jobID)

Description copied from interface: HighAvailabilityServices

Gets the leader retriever for the job JobMaster which is responsible for the given job.

Specified by:

getJobManagerLeaderRetriever in interface HighAvailabilityServices

Parameters:

jobID - The identifier of the job.

Returns:

Leader retrieval service to retrieve the job manager for the given job

getJobManagerLeaderRetriever

public LeaderRetrievalService getJobManagerLeaderRetriever(JobID jobID,
                                                           String defaultJobManagerAddress)

Description copied from interface: HighAvailabilityServices

Gets the leader retriever for the job JobMaster which is responsible for the given job.

Specified by:

getJobManagerLeaderRetriever in interface HighAvailabilityServices

Parameters:

jobID - The identifier of the job.

defaultJobManagerAddress - JobManager address which will be returned by a static leader retrieval service.

Returns:

Leader retrieval service to retrieve the job manager for the given job

getClusterRestEndpointLeaderRetriever

public LeaderRetrievalService getClusterRestEndpointLeaderRetriever()

Description copied from interface: ClientHighAvailabilityServices

Get the leader retriever for the cluster's rest endpoint.

Specified by:

getClusterRestEndpointLeaderRetriever in interface ClientHighAvailabilityServices

Specified by:

getClusterRestEndpointLeaderRetriever in interface HighAvailabilityServices

Returns:

the leader retriever for cluster's rest endpoint.

getResourceManagerLeaderElectionService

public LeaderElectionService getResourceManagerLeaderElectionService()

Description copied from interface: HighAvailabilityServices

Gets the leader election service for the cluster's resource manager.

Specified by:

getResourceManagerLeaderElectionService in interface HighAvailabilityServices

Returns:

Leader election service for the resource manager leader election

getDispatcherLeaderElectionService

public LeaderElectionService getDispatcherLeaderElectionService()

Description copied from interface: HighAvailabilityServices

Gets the leader election service for the cluster's dispatcher.

Specified by:

getDispatcherLeaderElectionService in interface HighAvailabilityServices

Returns:

Leader election service for the dispatcher leader election

getJobManagerLeaderElectionService

public LeaderElectionService getJobManagerLeaderElectionService(JobID jobID)

Description copied from interface: HighAvailabilityServices

Gets the leader election service for the given job.

Specified by:

getJobManagerLeaderElectionService in interface HighAvailabilityServices

Parameters:

jobID - The identifier of the job running the election.

Returns:

Leader election service for the job manager leader election

getClusterRestEndpointLeaderElectionService

public LeaderElectionService getClusterRestEndpointLeaderElectionService()

Description copied from interface: HighAvailabilityServices

Gets the leader election service for the cluster's rest endpoint.

Specified by:

getClusterRestEndpointLeaderElectionService in interface HighAvailabilityServices

Returns:

the leader election service used by the cluster's rest endpoint

getCheckpointRecoveryFactory

public CheckpointRecoveryFactory getCheckpointRecoveryFactory()

Description copied from interface: HighAvailabilityServices

Gets the checkpoint recovery factory for the job manager.

Specified by:

getCheckpointRecoveryFactory in interface HighAvailabilityServices

Returns:

Checkpoint recovery factory

getJobGraphStore

public JobGraphStore getJobGraphStore()
                               throws Exception

Description copied from interface: HighAvailabilityServices

Gets the submitted job graph store for the job manager.

Specified by:

getJobGraphStore in interface HighAvailabilityServices

Returns:

Submitted job graph store

Throws:

Exception - if the submitted job graph store could not be created

getRunningJobsRegistry

public RunningJobsRegistry getRunningJobsRegistry()

Description copied from interface: HighAvailabilityServices

Gets the registry that holds information about whether jobs are currently running.

Specified by:

getRunningJobsRegistry in interface HighAvailabilityServices

Returns:

Running job registry to retrieve running jobs

createBlobStore

public BlobStore createBlobStore()
                          throws IOException

Description copied from interface: HighAvailabilityServices

Creates the BLOB store in which BLOBs are stored in a highly-available fashion.

Specified by:

createBlobStore in interface HighAvailabilityServices

Returns:

Blob store

Throws:

IOException - if the blob store could not be created

close

public void close()
           throws Exception

Description copied from interface: HighAvailabilityServices

Closes the high availability services, releasing all resources.

This method does not delete or clean up any data stored in external stores (file systems, ZooKeeper, etc). Another instance of the high availability services will be able to recover the job.

If an exception occurs during closing services, this method will attempt to continue closing other services and report exceptions only after all services have been attempted to be closed.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface HighAvailabilityServices

Throws:

Exception - Thrown, if an exception occurred while closing these services.

closeAndCleanupAllData

public void closeAndCleanupAllData()
                            throws Exception

Description copied from interface: HighAvailabilityServices

Closes the high availability services (releasing all resources) and deletes all data stored by these services in external stores.

After this method was called, the any job or session that was managed by these high availability services will be unrecoverable.

If an exception occurs during cleanup, this method will attempt to continue the cleanup and report exceptions only after all cleanup steps have been attempted.

Specified by:

closeAndCleanupAllData in interface HighAvailabilityServices

Throws:

Exception - Thrown, if an exception occurred while closing these services or cleaning up data stored by them.