org.apache.flink.kubernetes

Class KubernetesResourceManager

java.lang.Object

org.apache.flink.runtime.rpc.RpcEndpoint

org.apache.flink.runtime.rpc.FencedRpcEndpoint<ResourceManagerId>

org.apache.flink.runtime.resourcemanager.ResourceManager<WorkerType>

org.apache.flink.runtime.resourcemanager.ActiveResourceManager<KubernetesWorkerNode>

org.apache.flink.kubernetes.KubernetesResourceManager

All Implemented Interfaces:

AutoCloseable, FlinkKubeClient.PodCallbackHandler, ClusterPartitionManager, LeaderContender, ResourceManagerGateway, FencedRpcGateway<ResourceManagerId>, RpcGateway, AutoCloseableAsync

public class KubernetesResourceManager
extends ActiveResourceManager<KubernetesWorkerNode>
implements FlinkKubeClient.PodCallbackHandler

Kubernetes specific implementation of the ResourceManager.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.runtime.resourcemanager.ActiveResourceManager

ActiveResourceManager.PendingWorkerNums

Nested classes/interfaces inherited from class org.apache.flink.runtime.rpc.RpcEndpoint

RpcEndpoint.MainThreadExecutor

Field Summary

Fields inherited from class org.apache.flink.runtime.resourcemanager.ActiveResourceManager

env, flinkClientConfig, flinkConfig

Fields inherited from class org.apache.flink.runtime.resourcemanager.ResourceManager

RESOURCE_MANAGER_NAME

Fields inherited from class org.apache.flink.runtime.rpc.RpcEndpoint

log, rpcServer

Constructor Summary

Constructors

Constructor and Description

KubernetesResourceManager(RpcService rpcService, ResourceID resourceId, Configuration flinkConfig, HighAvailabilityServices highAvailabilityServices, HeartbeatServices heartbeatServices, SlotManager slotManager, ResourceManagerPartitionTrackerFactory clusterPartitionTrackerFactory, JobLeaderIdService jobLeaderIdService, ClusterInformation clusterInformation, FatalErrorHandler fatalErrorHandler, ResourceManagerMetricGroup resourceManagerMetricGroup, FlinkKubeClient kubeClient, KubernetesResourceManagerConfiguration configuration)

Method Summary

Modifier and Type	Method and Description
void	handleFatalError(Throwable throwable)
protected void	initialize() Initializes the framework specific components.
protected void	internalDeregisterApplication(ApplicationStatus finalStatus, String diagnostics) The framework specific code to deregister the application.
protected Configuration	loadClientConfiguration()
void	onAdded(List<KubernetesPod> pods)
void	onDeleted(List<KubernetesPod> pods)
void	onError(List<KubernetesPod> pods)
void	onModified(List<KubernetesPod> pods)
CompletableFuture<Void>	onStop() User overridable callback which is called from RpcEndpoint.internalCallOnStop().
boolean	startNewWorker(WorkerResourceSpec workerResourceSpec) Allocates a resource using the worker resource specification.
boolean	stopWorker(KubernetesWorkerNode worker) Stops the given worker.
protected KubernetesWorkerNode	workerStarted(ResourceID resourceID) Callback when a worker was started.

Methods inherited from class org.apache.flink.runtime.resourcemanager.ActiveResourceManager

getNumRequestedNotAllocatedWorkers, getNumRequestedNotAllocatedWorkersFor, getNumRequestedNotRegisteredWorkers, getNumRequestedNotRegisteredWorkersFor, getStopTerminationFutureOrCompletedExceptionally, notifyAllocatedWorkerStopped, notifyNewWorkerAllocated, notifyNewWorkerAllocationFailed, notifyNewWorkerRequested, onTaskManagerRegistration

Methods inherited from class org.apache.flink.runtime.resourcemanager.ResourceManager

cancelSlotRequest, clearStateAsync, closeJobManagerConnection, closeTaskManagerConnection, deregisterApplication, disconnectJobManager, disconnectTaskManager, getNumberOfRegisteredTaskManagers, getNumberRequiredTaskManagers, getRequiredResources, grantLeadership, handleError, heartbeatFromJobManager, heartbeatFromTaskManager, jobLeaderLostLeadership, listDataSets, notifySlotAvailable, onFatalError, onStart, prepareLeadershipAsync, registerJobManager, registerTaskExecutor, releaseClusterPartitions, releaseResource, removeJob, requestResourceOverview, requestSlot, requestTaskManagerFileUploadByName, requestTaskManagerFileUploadByType, requestTaskManagerInfo, requestTaskManagerInfo, requestTaskManagerLogList, requestTaskManagerMetricQueryServiceAddresses, requestThreadDump, revokeLeadership, sendSlotReport, setFailUnfulfillableRequest, startServicesOnLeadership

Methods inherited from class org.apache.flink.runtime.rpc.FencedRpcEndpoint

callAsyncWithoutFencing, getFencingToken, getMainThreadExecutor, getUnfencedMainThreadExecutor, runAsyncWithoutFencing, setFencingToken

Methods inherited from class org.apache.flink.runtime.rpc.RpcEndpoint

callAsync, closeAsync, getAddress, getEndpointId, getHostname, getRpcService, getSelfGateway, getTerminationFuture, internalCallOnStart, internalCallOnStop, isRunning, runAsync, scheduleRunAsync, scheduleRunAsync, start, stop, validateRunsInMainThread

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.rpc.FencedRpcGateway

getFencingToken

Methods inherited from interface org.apache.flink.runtime.rpc.RpcGateway

getAddress, getHostname

Methods inherited from interface org.apache.flink.runtime.leaderelection.LeaderContender

getDescription

Methods inherited from interface org.apache.flink.util.AutoCloseableAsync

close

Constructor Detail

KubernetesResourceManager

public KubernetesResourceManager(RpcService rpcService,
                                 ResourceID resourceId,
                                 Configuration flinkConfig,
                                 HighAvailabilityServices highAvailabilityServices,
                                 HeartbeatServices heartbeatServices,
                                 SlotManager slotManager,
                                 ResourceManagerPartitionTrackerFactory clusterPartitionTrackerFactory,
                                 JobLeaderIdService jobLeaderIdService,
                                 ClusterInformation clusterInformation,
                                 FatalErrorHandler fatalErrorHandler,
                                 ResourceManagerMetricGroup resourceManagerMetricGroup,
                                 FlinkKubeClient kubeClient,
                                 KubernetesResourceManagerConfiguration configuration)

Method Detail

loadClientConfiguration

protected Configuration loadClientConfiguration()

Specified by:

loadClientConfiguration in class ActiveResourceManager<KubernetesWorkerNode>

initialize

protected void initialize()
                   throws ResourceManagerException

Description copied from class: ResourceManager

Initializes the framework specific components.

Specified by:

initialize in class ResourceManager<KubernetesWorkerNode>

Throws:

ResourceManagerException - which occurs during initialization and causes the resource manager to fail.

onStop

public CompletableFuture<Void> onStop()

Description copied from class: RpcEndpoint

User overridable callback which is called from RpcEndpoint.internalCallOnStop().

This method is called when the RpcEndpoint is being shut down. The method is guaranteed to be executed in the main thread context and can be used to clean up internal state.

IMPORTANT: This method should never be called directly by the user.

Overrides:

onStop in class ResourceManager<KubernetesWorkerNode>

Returns:

Future which is completed once all post stop actions are completed. If an error occurs this future is completed exceptionally

internalDeregisterApplication

protected void internalDeregisterApplication(ApplicationStatus finalStatus,
                                             @Nullable
                                             String diagnostics)

Description copied from class: ResourceManager

The framework specific code to deregister the application. This should report the application's final status and shut down the resource manager cleanly.

This method also needs to make sure all pending containers that are not registered yet are returned.

Specified by:

internalDeregisterApplication in class ResourceManager<KubernetesWorkerNode>

Parameters:

finalStatus - The application status to report.

diagnostics - A diagnostics message or null.

startNewWorker

public boolean startNewWorker(WorkerResourceSpec workerResourceSpec)

Description copied from class: ResourceManager

Allocates a resource using the worker resource specification.

Specified by:

startNewWorker in class ResourceManager<KubernetesWorkerNode>

Parameters:

workerResourceSpec - workerResourceSpec specifies the size of the to be allocated resource

Returns:

whether the resource can be allocated

workerStarted

protected KubernetesWorkerNode workerStarted(ResourceID resourceID)

Description copied from class: ResourceManager

Callback when a worker was started.

Specified by:

workerStarted in class ResourceManager<KubernetesWorkerNode>

Parameters:

resourceID - The worker resource id

stopWorker

public boolean stopWorker(KubernetesWorkerNode worker)

Description copied from class: ResourceManager

Stops the given worker.

Specified by:

stopWorker in class ResourceManager<KubernetesWorkerNode>

Parameters:

worker - The worker.

Returns:

True if the worker was stopped, otherwise false

onAdded

public void onAdded(List<KubernetesPod> pods)

Specified by:

onAdded in interface FlinkKubeClient.PodCallbackHandler

onModified

public void onModified(List<KubernetesPod> pods)

Specified by:

onModified in interface FlinkKubeClient.PodCallbackHandler

onDeleted

public void onDeleted(List<KubernetesPod> pods)

Specified by:

onDeleted in interface FlinkKubeClient.PodCallbackHandler

onError

public void onError(List<KubernetesPod> pods)

Specified by:

onError in interface FlinkKubeClient.PodCallbackHandler

handleFatalError

public void handleFatalError(Throwable throwable)

Specified by:

handleFatalError in interface FlinkKubeClient.PodCallbackHandler