org.apache.flink.mesos.runtime.clusterframework

Class MesosResourceManager

java.lang.Object

org.apache.flink.runtime.rpc.RpcEndpoint

org.apache.flink.runtime.rpc.FencedRpcEndpoint<ResourceManagerId>

org.apache.flink.runtime.resourcemanager.ResourceManager<RegisteredMesosWorkerNode>

org.apache.flink.mesos.runtime.clusterframework.MesosResourceManager

All Implemented Interfaces:

LeaderContender, ResourceManagerGateway, FencedRpcGateway<ResourceManagerId>, RpcGateway

public class MesosResourceManager
extends ResourceManager<RegisteredMesosWorkerNode>

The Mesos implementation of the resource manager.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.runtime.rpc.RpcEndpoint

RpcEndpoint.MainThreadExecutor

Field Summary

Fields

Modifier and Type	Field and Description
protected static org.slf4j.Logger	LOG

Fields inherited from class org.apache.flink.runtime.resourcemanager.ResourceManager

RESOURCE_MANAGER_NAME

Fields inherited from class org.apache.flink.runtime.rpc.RpcEndpoint

log, rpcServer

Constructor Summary

Constructors

Constructor and Description

MesosResourceManager(RpcService rpcService, String resourceManagerEndpointId, ResourceID resourceId, ResourceManagerConfiguration resourceManagerConfiguration, HighAvailabilityServices highAvailabilityServices, HeartbeatServices heartbeatServices, SlotManager slotManager, MetricRegistry metricRegistry, JobLeaderIdService jobLeaderIdService, ClusterInformation clusterInformation, FatalErrorHandler fatalErrorHandler, Configuration flinkConfig, MesosServices mesosServices, MesosConfiguration mesosConfig, MesosTaskManagerParameters taskManagerParameters, ContainerSpecification taskManagerContainerSpec, String webUiUrl, JobManagerMetricGroup jobManagerMetricGroup)

Method Summary

Modifier and Type	Method and Description
void	acceptOffers(AcceptOffers msg) Accept offers as advised by the launch coordinator.
protected CompletableFuture<Void>	clearStateAsync() This method can be overridden to add a (non-blocking) state clearing routine to the ResourceManager that will be called when leadership is revoked.
protected akka.actor.ActorRef	createConnectionMonitor()
protected akka.actor.ActorRef	createLaunchCoordinator(org.apache.mesos.SchedulerDriver schedulerDriver, akka.actor.ActorRef selfActor)
protected akka.actor.ActorRef	createReconciliationCoordinator(org.apache.mesos.SchedulerDriver schedulerDriver)
protected akka.actor.ActorRef	createSelfActor()
protected akka.actor.ActorRef	createTaskMonitor(org.apache.mesos.SchedulerDriver schedulerDriver)
protected void	disconnected(Disconnected message) Called when disconnected from Mesos.
protected void	executorLost(ExecutorLost message)
protected void	frameworkMessage(FrameworkMessage message)
protected void	initialize() Initializes the framework specific components.
protected void	internalDeregisterApplication(ApplicationStatus finalStatus, String diagnostics) The framework specific code to deregister the application.
protected void	offerRescinded(OfferRescinded message) Called when resource offers are rescinded.
CompletableFuture<Void>	postStop() User overridable callback.
protected CompletableFuture<Void>	prepareLeadershipAsync() This method can be overridden to add a (non-blocking) initialization routine to the ResourceManager that will be called when leadership is granted but before leadership is confirmed.
void	reconcile(org.apache.flink.mesos.scheduler.ReconciliationCoordinator.Reconcile message) Handles a reconciliation request from a task monitor.
protected void	registered(Registered message)
protected void	reregistered(ReRegistered message) Called when reconnected to Mesos following a failover event.
protected void	resourceOffers(ResourceOffers message) Called when resource offers are made to the framework.
protected void	slaveLost(SlaveLost message)
void	startNewWorker(ResourceProfile resourceProfile) Allocates a resource using the resource profile.
protected void	statusUpdate(StatusUpdate message) Handles a task status update from Mesos.
boolean	stopWorker(RegisteredMesosWorkerNode workerNode) Stops the given worker.
void	taskTerminated(org.apache.flink.mesos.scheduler.TaskMonitor.TaskTerminated message) Handles a termination notification from a task monitor.
protected RegisteredMesosWorkerNode	workerStarted(ResourceID resourceID) Callback when a worker was started.

Methods inherited from class org.apache.flink.runtime.resourcemanager.ResourceManager

cancelSlotRequest, closeJobManagerConnection, closeTaskManagerConnection, deregisterApplication, disconnectJobManager, disconnectTaskManager, getNumberOfRegisteredTaskManagers, getNumberPendingSlotRequests, grantLeadership, handleError, heartbeatFromJobManager, heartbeatFromTaskManager, jobLeaderLostLeadership, notifySlotAvailable, onFatalError, registerInfoMessageListener, registerJobManager, registerTaskExecutor, releaseResource, removeJob, requestResourceOverview, requestSlot, requestTaskManagerFileUpload, requestTaskManagerInfo, requestTaskManagerInfo, requestTaskManagerMetricQueryServicePaths, revokeLeadership, sendInfoMessage, sendSlotReport, start, unRegisterInfoMessageListener

Methods inherited from class org.apache.flink.runtime.rpc.FencedRpcEndpoint

callAsyncWithoutFencing, getFencingToken, getMainThreadExecutor, getUnfencedMainThreadExecutor, runAsyncWithoutFencing, setFencingToken

Methods inherited from class org.apache.flink.runtime.rpc.RpcEndpoint

callAsync, getAddress, getEndpointId, getHostname, getRpcService, getSelfGateway, getTerminationFuture, runAsync, scheduleRunAsync, scheduleRunAsync, shutDown, stop, validateRunsInMainThread

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.rpc.FencedRpcGateway

getFencingToken

Methods inherited from interface org.apache.flink.runtime.rpc.RpcGateway

getAddress, getHostname

Methods inherited from interface org.apache.flink.runtime.leaderelection.LeaderContender

getAddress

Field Detail

LOG

protected static final org.slf4j.Logger LOG

Constructor Detail

MesosResourceManager

public MesosResourceManager(RpcService rpcService,
                            String resourceManagerEndpointId,
                            ResourceID resourceId,
                            ResourceManagerConfiguration resourceManagerConfiguration,
                            HighAvailabilityServices highAvailabilityServices,
                            HeartbeatServices heartbeatServices,
                            SlotManager slotManager,
                            MetricRegistry metricRegistry,
                            JobLeaderIdService jobLeaderIdService,
                            ClusterInformation clusterInformation,
                            FatalErrorHandler fatalErrorHandler,
                            Configuration flinkConfig,
                            MesosServices mesosServices,
                            MesosConfiguration mesosConfig,
                            MesosTaskManagerParameters taskManagerParameters,
                            ContainerSpecification taskManagerContainerSpec,
                            @Nullable
                            String webUiUrl,
                            JobManagerMetricGroup jobManagerMetricGroup)

Method Detail

createSelfActor

protected akka.actor.ActorRef createSelfActor()

createConnectionMonitor

protected akka.actor.ActorRef createConnectionMonitor()

createTaskMonitor

protected akka.actor.ActorRef createTaskMonitor(org.apache.mesos.SchedulerDriver schedulerDriver)

createLaunchCoordinator

protected akka.actor.ActorRef createLaunchCoordinator(org.apache.mesos.SchedulerDriver schedulerDriver,
                                                      akka.actor.ActorRef selfActor)

createReconciliationCoordinator

protected akka.actor.ActorRef createReconciliationCoordinator(org.apache.mesos.SchedulerDriver schedulerDriver)

initialize

protected void initialize()
                   throws ResourceManagerException

Description copied from class: ResourceManager

Initializes the framework specific components.

Specified by:

initialize in class ResourceManager<RegisteredMesosWorkerNode>

Throws:

ResourceManagerException - which occurs during initialization and causes the resource manager to fail.

prepareLeadershipAsync

protected CompletableFuture<Void> prepareLeadershipAsync()

Description copied from class: ResourceManager

This method can be overridden to add a (non-blocking) initialization routine to the ResourceManager that will be called when leadership is granted but before leadership is confirmed.

Overrides:

prepareLeadershipAsync in class ResourceManager<RegisteredMesosWorkerNode>

Returns:

Returns a CompletableFuture that completes when the computation is finished.

clearStateAsync

protected CompletableFuture<Void> clearStateAsync()

Description copied from class: ResourceManager

This method can be overridden to add a (non-blocking) state clearing routine to the ResourceManager that will be called when leadership is revoked.

Overrides:

clearStateAsync in class ResourceManager<RegisteredMesosWorkerNode>

Returns:

Returns a CompletableFuture that completes when the state clearing routine is finished.

postStop

public CompletableFuture<Void> postStop()

Description copied from class: RpcEndpoint

User overridable callback.

This method is called when the RpcEndpoint is being shut down. The method is guaranteed to be executed in the main thread context and can be used to clean up internal state.

IMPORTANT: This method should never be called directly by the user.

Overrides:

postStop in class ResourceManager<RegisteredMesosWorkerNode>

Returns:

Future which is completed once all post stop actions are completed. If an error occurs this future is completed exceptionally

internalDeregisterApplication

protected void internalDeregisterApplication(ApplicationStatus finalStatus,
                                             @Nullable
                                             String diagnostics)
                                      throws ResourceManagerException

Description copied from class: ResourceManager

The framework specific code to deregister the application. This should report the application's final status and shut down the resource manager cleanly.

This method also needs to make sure all pending containers that are not registered yet are returned.

Specified by:

internalDeregisterApplication in class ResourceManager<RegisteredMesosWorkerNode>

Parameters:

finalStatus - The application status to report.

diagnostics - A diagnostics message or null.

Throws:

ResourceManagerException - if the application could not be shut down.

startNewWorker

public void startNewWorker(ResourceProfile resourceProfile)

Description copied from class: ResourceManager

Allocates a resource using the resource profile.

Specified by:

startNewWorker in class ResourceManager<RegisteredMesosWorkerNode>

Parameters:

resourceProfile - The resource description

stopWorker

public boolean stopWorker(RegisteredMesosWorkerNode workerNode)

Description copied from class: ResourceManager

Stops the given worker.

Specified by:

stopWorker in class ResourceManager<RegisteredMesosWorkerNode>

Parameters:

workerNode - The worker.

Returns:

True if the worker was stopped, otherwise false

workerStarted

protected RegisteredMesosWorkerNode workerStarted(ResourceID resourceID)

Callback when a worker was started.

Specified by:

workerStarted in class ResourceManager<RegisteredMesosWorkerNode>

Parameters:

resourceID - The worker resource id (as provided by the TaskExecutor)

registered

protected void registered(Registered message)

reregistered

protected void reregistered(ReRegistered message)

Called when reconnected to Mesos following a failover event.

disconnected

protected void disconnected(Disconnected message)

Called when disconnected from Mesos.

resourceOffers

protected void resourceOffers(ResourceOffers message)

Called when resource offers are made to the framework.

offerRescinded

protected void offerRescinded(OfferRescinded message)

Called when resource offers are rescinded.

statusUpdate

protected void statusUpdate(StatusUpdate message)

Handles a task status update from Mesos.

frameworkMessage

protected void frameworkMessage(FrameworkMessage message)

slaveLost

protected void slaveLost(SlaveLost message)

executorLost

protected void executorLost(ExecutorLost message)

acceptOffers

public void acceptOffers(AcceptOffers msg)

Accept offers as advised by the launch coordinator.

Acceptance is routed through the RM to update the persistent state before forwarding the message to Mesos.

reconcile

public void reconcile(org.apache.flink.mesos.scheduler.ReconciliationCoordinator.Reconcile message)

Handles a reconciliation request from a task monitor.

taskTerminated

public void taskTerminated(org.apache.flink.mesos.scheduler.TaskMonitor.TaskTerminated message)

Handles a termination notification from a task monitor.