org.apache.flink.runtime.jobmaster

Class JobMaster

java.lang.Object

org.apache.flink.runtime.rpc.RpcEndpoint

org.apache.flink.runtime.rpc.FencedRpcEndpoint<JobMasterId>

org.apache.flink.runtime.jobmaster.JobMaster

All Implemented Interfaces:

CheckpointCoordinatorGateway, JobMasterGateway, KvStateLocationOracle, KvStateRegistryGateway, FencedRpcGateway<JobMasterId>, RpcGateway

public class JobMaster
extends FencedRpcEndpoint<JobMasterId>
implements JobMasterGateway

JobMaster implementation. The job master is responsible for the execution of a single JobGraph.

It offers the following methods as part of its rpc interface to interact with the JobMaster remotely:

• updateTaskExecutionState(org.apache.flink.runtime.taskmanager.TaskExecutionState) updates the task execution state for given task

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.runtime.rpc.RpcEndpoint

RpcEndpoint.MainThreadExecutor

Field Summary

Fields

Modifier and Type	Field and Description
static String	ARCHIVE_NAME
static String	JOB_MANAGER_NAME Default names for Flink's distributed components.

Fields inherited from class org.apache.flink.runtime.rpc.RpcEndpoint

log, rpcServer

Constructor Summary

Constructors

Constructor and Description

JobMaster(RpcService rpcService, JobMasterConfiguration jobMasterConfiguration, ResourceID resourceId, JobGraph jobGraph, HighAvailabilityServices highAvailabilityService, SlotPoolFactory slotPoolFactory, JobManagerSharedServices jobManagerSharedServices, HeartbeatServices heartbeatServices, BlobServer blobServer, JobManagerJobMetricGroupFactory jobMetricGroupFactory, OnCompletionActions jobCompletionActions, FatalErrorHandler fatalErrorHandler, ClassLoader userCodeLoader)

Method Summary

Modifier and Type	Method and Description
void	acknowledgeCheckpoint(JobID jobID, ExecutionAttemptID executionAttemptID, long checkpointId, CheckpointMetrics checkpointMetrics, TaskStateSnapshot checkpointState)
CompletableFuture<Acknowledge>	cancel(Time timeout) Cancels the currently executed job.
void	declineCheckpoint(DeclineCheckpoint decline)
void	disconnectResourceManager(ResourceManagerId resourceManagerId, Exception cause) Disconnects the resource manager from the job manager because of the given cause.
CompletableFuture<Acknowledge>	disconnectTaskManager(ResourceID resourceID, Exception cause) Disconnects the given TaskExecutor from the JobMaster.
void	failSlot(ResourceID taskManagerId, AllocationID allocationId, Exception cause) Fails the slot with the given allocation id and cause.
void	heartbeatFromResourceManager(ResourceID resourceID) Sends heartbeat request from the resource manager.
void	heartbeatFromTaskManager(ResourceID resourceID, AccumulatorReport accumulatorReport) Sends the heartbeat to job manager from task manager.
void	notifyAllocationFailure(AllocationID allocationID, Exception cause) Notifies that the allocation has failed.
CompletableFuture<Acknowledge>	notifyKvStateRegistered(JobID jobId, JobVertexID jobVertexId, KeyGroupRange keyGroupRange, String registrationName, KvStateID kvStateId, InetSocketAddress kvStateServerAddress) Notifies that queryable state has been registered.
CompletableFuture<Acknowledge>	notifyKvStateUnregistered(JobID jobId, JobVertexID jobVertexId, KeyGroupRange keyGroupRange, String registrationName) Notifies that queryable state has been unregistered.
CompletableFuture<Collection<SlotOffer>>	offerSlots(ResourceID taskManagerId, Collection<SlotOffer> slots, Time timeout) Offers the given slots to the job manager.
CompletableFuture<Void>	postStop() Suspend the job and shutdown all other services including rpc.
CompletableFuture<RegistrationResponse>	registerTaskManager(String taskManagerRpcAddress, TaskManagerLocation taskManagerLocation, Time timeout) Registers the task manager at the job manager.
CompletableFuture<ClassloadingProps>	requestClassloadingProps() Request the classloading props of this job.
CompletableFuture<ArchivedExecutionGraph>	requestJob(Time timeout) Requests the ArchivedExecutionGraph of the executed job.
CompletableFuture<JobDetails>	requestJobDetails(Time timeout) Request the details of the executed job.
CompletableFuture<JobStatus>	requestJobStatus(Time timeout) Requests the current job status.
CompletableFuture<KvStateLocation>	requestKvStateLocation(JobID jobId, String registrationName) Requests a KvStateLocation for the specified InternalKvState registration name.
CompletableFuture<SerializedInputSplit>	requestNextInputSplit(JobVertexID vertexID, ExecutionAttemptID executionAttempt) Requests the next input split for the ExecutionJobVertex.
CompletableFuture<OperatorBackPressureStatsResponse>	requestOperatorBackPressureStats(JobVertexID jobVertexId) Requests the statistics on operator back pressure.
CompletableFuture<ExecutionState>	requestPartitionState(IntermediateDataSetID intermediateResultId, ResultPartitionID resultPartitionId) Requests the current state of the partition.
CompletableFuture<Acknowledge>	rescaleJob(int newParallelism, RescalingBehaviour rescalingBehaviour, Time timeout) Triggers rescaling of the executed job.
CompletableFuture<Acknowledge>	rescaleOperators(Collection<JobVertexID> operators, int newParallelism, RescalingBehaviour rescalingBehaviour, Time timeout) Triggers rescaling of the given set of operators.
CompletableFuture<Acknowledge>	scheduleOrUpdateConsumers(ResultPartitionID partitionID, Time timeout) Notifies the JobManager about available data for a produced partition.
void	start() Starts the rpc endpoint.
CompletableFuture<Acknowledge>	start(JobMasterId newJobMasterId, Time timeout) Start the rpc service and begin to run the job.
CompletableFuture<Acknowledge>	stop(Time timeout) Cancel the currently executed job.
CompletableFuture<Acknowledge>	suspend(Exception cause, Time timeout) Suspending job, all the running tasks will be cancelled, and communication with other components will be disposed.
CompletableFuture<String>	triggerSavepoint(String targetDirectory, boolean cancelJob, Time timeout) Triggers taking a savepoint of the executed job.
CompletableFuture<Acknowledge>	updateTaskExecutionState(TaskExecutionState taskExecutionState) Updates the task execution state for a given task.

Methods inherited from class org.apache.flink.runtime.rpc.FencedRpcEndpoint

callAsyncWithoutFencing, getFencingToken, getMainThreadExecutor, getUnfencedMainThreadExecutor, runAsyncWithoutFencing, setFencingToken

Methods inherited from class org.apache.flink.runtime.rpc.RpcEndpoint

callAsync, getAddress, getEndpointId, getHostname, getRpcService, getSelfGateway, getTerminationFuture, runAsync, scheduleRunAsync, scheduleRunAsync, shutDown, stop, validateRunsInMainThread

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.rpc.FencedRpcGateway

getFencingToken

Methods inherited from interface org.apache.flink.runtime.rpc.RpcGateway

getAddress, getHostname

Field Detail

JOB_MANAGER_NAME

public static final String JOB_MANAGER_NAME

Default names for Flink's distributed components.

See Also:

Constant Field Values

ARCHIVE_NAME

public static final String ARCHIVE_NAME

See Also:

Constant Field Values

Constructor Detail

JobMaster

public JobMaster(RpcService rpcService,
                 JobMasterConfiguration jobMasterConfiguration,
                 ResourceID resourceId,
                 JobGraph jobGraph,
                 HighAvailabilityServices highAvailabilityService,
                 SlotPoolFactory slotPoolFactory,
                 JobManagerSharedServices jobManagerSharedServices,
                 HeartbeatServices heartbeatServices,
                 BlobServer blobServer,
                 JobManagerJobMetricGroupFactory jobMetricGroupFactory,
                 OnCompletionActions jobCompletionActions,
                 FatalErrorHandler fatalErrorHandler,
                 ClassLoader userCodeLoader)
          throws Exception

Throws:

Exception

Method Detail

start

public void start()

Description copied from class: RpcEndpoint

Starts the rpc endpoint. This tells the underlying rpc server that the rpc endpoint is ready to process remote procedure calls.

IMPORTANT: Whenever you override this method, call the parent implementation to enable rpc processing. It is advised to make the parent call last.

Overrides:

start in class RpcEndpoint

start

public CompletableFuture<Acknowledge> start(JobMasterId newJobMasterId,
                                            Time timeout)
                                     throws Exception

Start the rpc service and begin to run the job.

Parameters:

newJobMasterId - The necessary fencing token to run the job

timeout - for the operation

Returns:

Future acknowledge if the job could be started. Otherwise the future contains an exception

Throws:

Exception

suspend

public CompletableFuture<Acknowledge> suspend(Exception cause,
                                              Time timeout)

Suspending job, all the running tasks will be cancelled, and communication with other components will be disposed.

Mostly job is suspended because of the leadership has been revoked, one can be restart this job by calling the start(JobMasterId, Time) method once we take the leadership back again.

This method is executed asynchronously

Parameters:

cause - The reason of why this job been suspended.

timeout - for this operation

Returns:

Future acknowledge indicating that the job has been suspended. Otherwise the future contains an exception

postStop

public CompletableFuture<Void> postStop()

Suspend the job and shutdown all other services including rpc.

Specified by:

postStop in class RpcEndpoint

Returns:

Future which is completed once all post stop actions are completed. If an error occurs this future is completed exceptionally

cancel

public CompletableFuture<Acknowledge> cancel(Time timeout)

Description copied from interface: JobMasterGateway

Cancels the currently executed job.

Specified by:

cancel in interface JobMasterGateway

Parameters:

timeout - of this operation

Returns:

Future acknowledge of the operation

stop

public CompletableFuture<Acknowledge> stop(Time timeout)

Description copied from interface: JobMasterGateway

Cancel the currently executed job.

Specified by:

stop in interface JobMasterGateway

Parameters:

timeout - of this operation

Returns:

Future acknowledge if the cancellation was successful

rescaleJob

public CompletableFuture<Acknowledge> rescaleJob(int newParallelism,
                                                 RescalingBehaviour rescalingBehaviour,
                                                 Time timeout)

Description copied from interface: JobMasterGateway

Triggers rescaling of the executed job.

Specified by:

rescaleJob in interface JobMasterGateway

Parameters:

newParallelism - new parallelism of the job

rescalingBehaviour - defining how strict the rescaling has to be executed

timeout - of this operation

Returns:

Future which is completed with Acknowledge once the rescaling was successful

rescaleOperators

public CompletableFuture<Acknowledge> rescaleOperators(Collection<JobVertexID> operators,
                                                       int newParallelism,
                                                       RescalingBehaviour rescalingBehaviour,
                                                       Time timeout)

Description copied from interface: JobMasterGateway

Triggers rescaling of the given set of operators.

Specified by:

rescaleOperators in interface JobMasterGateway

Parameters:

operators - set of operators which shall be rescaled

newParallelism - new parallelism of the given set of operators

rescalingBehaviour - defining how strict the rescaling has to be executed

timeout - of this operation

Returns:

Future which is completed with Acknowledge once the rescaling was successful

updateTaskExecutionState

public CompletableFuture<Acknowledge> updateTaskExecutionState(TaskExecutionState taskExecutionState)

Updates the task execution state for a given task.

Specified by:

updateTaskExecutionState in interface JobMasterGateway

Parameters:

taskExecutionState - New task execution state for a given task

Returns:

Acknowledge the task execution state update

requestNextInputSplit

public CompletableFuture<SerializedInputSplit> requestNextInputSplit(JobVertexID vertexID,
                                                                     ExecutionAttemptID executionAttempt)

Description copied from interface: JobMasterGateway

Requests the next input split for the ExecutionJobVertex. The next input split is sent back to the sender as a SerializedInputSplit message.

Specified by:

requestNextInputSplit in interface JobMasterGateway

Parameters:

vertexID - The job vertex id

executionAttempt - The execution attempt id

Returns:

The future of the input split. If there is no further input split, will return an empty object.

requestPartitionState

public CompletableFuture<ExecutionState> requestPartitionState(IntermediateDataSetID intermediateResultId,
                                                               ResultPartitionID resultPartitionId)

Description copied from interface: JobMasterGateway

Requests the current state of the partition. The state of a partition is currently bound to the state of the producing execution.

Specified by:

requestPartitionState in interface JobMasterGateway

Parameters:

intermediateResultId - The execution attempt ID of the task requesting the partition state.

resultPartitionId - The partition ID of the partition to request the state of.

Returns:

The future of the partition state

scheduleOrUpdateConsumers

public CompletableFuture<Acknowledge> scheduleOrUpdateConsumers(ResultPartitionID partitionID,
                                                                Time timeout)

Description copied from interface: JobMasterGateway

Notifies the JobManager about available data for a produced partition.

There is a call to this method for each ExecutionVertex instance once per produced ResultPartition instance, either when first producing data (for pipelined executions) or when all data has been produced (for staged executions).

The JobManager then can decide when to schedule the partition consumers of the given session.

Specified by:

scheduleOrUpdateConsumers in interface JobMasterGateway

Parameters:

partitionID - The partition which has already produced data

timeout - before the rpc call fails

Returns:

Future acknowledge of the schedule or update operation

disconnectTaskManager

public CompletableFuture<Acknowledge> disconnectTaskManager(ResourceID resourceID,
                                                            Exception cause)

Description copied from interface: JobMasterGateway

Disconnects the given TaskExecutor from the JobMaster.

Specified by:

disconnectTaskManager in interface JobMasterGateway

Parameters:

resourceID - identifying the TaskManager to disconnect

cause - for the disconnection of the TaskManager

Returns:

Future acknowledge once the JobMaster has been disconnected from the TaskManager

acknowledgeCheckpoint

public void acknowledgeCheckpoint(JobID jobID,
                                  ExecutionAttemptID executionAttemptID,
                                  long checkpointId,
                                  CheckpointMetrics checkpointMetrics,
                                  TaskStateSnapshot checkpointState)

Specified by:

acknowledgeCheckpoint in interface CheckpointCoordinatorGateway

declineCheckpoint

public void declineCheckpoint(DeclineCheckpoint decline)

Specified by:

declineCheckpoint in interface CheckpointCoordinatorGateway

requestKvStateLocation

public CompletableFuture<KvStateLocation> requestKvStateLocation(JobID jobId,
                                                                 String registrationName)

Description copied from interface: KvStateLocationOracle

Requests a KvStateLocation for the specified InternalKvState registration name.

Specified by:

requestKvStateLocation in interface KvStateLocationOracle

Parameters:

jobId - identifying the job for which to request the KvStateLocation

registrationName - Name under which the KvState has been registered.

Returns:

Future of the requested InternalKvState location

notifyKvStateRegistered

public CompletableFuture<Acknowledge> notifyKvStateRegistered(JobID jobId,
                                                              JobVertexID jobVertexId,
                                                              KeyGroupRange keyGroupRange,
                                                              String registrationName,
                                                              KvStateID kvStateId,
                                                              InetSocketAddress kvStateServerAddress)

Description copied from interface: KvStateRegistryGateway

Notifies that queryable state has been registered.

Specified by:

notifyKvStateRegistered in interface KvStateRegistryGateway

Parameters:

jobId - identifying the job for which to register a key value state

jobVertexId - JobVertexID the KvState instance belongs to.

keyGroupRange - Key group range the KvState instance belongs to.

registrationName - Name under which the KvState has been registered.

kvStateId - ID of the registered KvState instance.

kvStateServerAddress - Server address where to find the KvState instance.

Returns:

Future acknowledge if the key-value state has been registered

notifyKvStateUnregistered

public CompletableFuture<Acknowledge> notifyKvStateUnregistered(JobID jobId,
                                                                JobVertexID jobVertexId,
                                                                KeyGroupRange keyGroupRange,
                                                                String registrationName)

Description copied from interface: KvStateRegistryGateway

Notifies that queryable state has been unregistered.

Specified by:

notifyKvStateUnregistered in interface KvStateRegistryGateway

Parameters:

jobId - identifying the job for which to unregister a key value state

jobVertexId - JobVertexID the KvState instance belongs to.

keyGroupRange - Key group index the KvState instance belongs to.

registrationName - Name under which the KvState has been registered.

Returns:

Future acknowledge if the key-value state has been unregistered

requestClassloadingProps

public CompletableFuture<ClassloadingProps> requestClassloadingProps()

Description copied from interface: JobMasterGateway

Request the classloading props of this job.

Specified by:

requestClassloadingProps in interface JobMasterGateway

offerSlots

public CompletableFuture<Collection<SlotOffer>> offerSlots(ResourceID taskManagerId,
                                                           Collection<SlotOffer> slots,
                                                           Time timeout)

Description copied from interface: JobMasterGateway

Offers the given slots to the job manager. The response contains the set of accepted slots.

Specified by:

offerSlots in interface JobMasterGateway

Parameters:

taskManagerId - identifying the task manager

slots - to offer to the job manager

timeout - for the rpc call

Returns:

Future set of accepted slots.

failSlot

public void failSlot(ResourceID taskManagerId,
                     AllocationID allocationId,
                     Exception cause)

Description copied from interface: JobMasterGateway

Fails the slot with the given allocation id and cause.

Specified by:

failSlot in interface JobMasterGateway

Parameters:

taskManagerId - identifying the task manager

allocationId - identifying the slot to fail

cause - of the failing

registerTaskManager

public CompletableFuture<RegistrationResponse> registerTaskManager(String taskManagerRpcAddress,
                                                                   TaskManagerLocation taskManagerLocation,
                                                                   Time timeout)

Description copied from interface: JobMasterGateway

Registers the task manager at the job manager.

Specified by:

registerTaskManager in interface JobMasterGateway

Parameters:

taskManagerRpcAddress - the rpc address of the task manager

taskManagerLocation - location of the task manager

timeout - for the rpc call

Returns:

Future registration response indicating whether the registration was successful or not

disconnectResourceManager

public void disconnectResourceManager(ResourceManagerId resourceManagerId,
                                      Exception cause)

Description copied from interface: JobMasterGateway

Disconnects the resource manager from the job manager because of the given cause.

Specified by:

disconnectResourceManager in interface JobMasterGateway

Parameters:

resourceManagerId - identifying the resource manager leader id

cause - of the disconnect

heartbeatFromTaskManager

public void heartbeatFromTaskManager(ResourceID resourceID,
                                     AccumulatorReport accumulatorReport)

Description copied from interface: JobMasterGateway

Sends the heartbeat to job manager from task manager.

Specified by:

heartbeatFromTaskManager in interface JobMasterGateway

Parameters:

resourceID - unique id of the task manager

accumulatorReport - report containing accumulator updates

heartbeatFromResourceManager

public void heartbeatFromResourceManager(ResourceID resourceID)

Description copied from interface: JobMasterGateway

Sends heartbeat request from the resource manager.

Specified by:

heartbeatFromResourceManager in interface JobMasterGateway

Parameters:

resourceID - unique id of the resource manager

requestJobDetails

public CompletableFuture<JobDetails> requestJobDetails(Time timeout)

Description copied from interface: JobMasterGateway

Request the details of the executed job.

Specified by:

requestJobDetails in interface JobMasterGateway

Parameters:

timeout - for the rpc call

Returns:

Future details of the executed job

requestJobStatus

public CompletableFuture<JobStatus> requestJobStatus(Time timeout)

Description copied from interface: JobMasterGateway

Requests the current job status.

Specified by:

requestJobStatus in interface JobMasterGateway

Parameters:

timeout - for the rpc call

Returns:

Future containing the current job status

requestJob

public CompletableFuture<ArchivedExecutionGraph> requestJob(Time timeout)

Description copied from interface: JobMasterGateway

Requests the ArchivedExecutionGraph of the executed job.

Specified by:

requestJob in interface JobMasterGateway

Parameters:

timeout - for the rpc call

Returns:

Future which is completed with the ArchivedExecutionGraph of the executed job

triggerSavepoint

public CompletableFuture<String> triggerSavepoint(@Nullable
                                                  String targetDirectory,
                                                  boolean cancelJob,
                                                  Time timeout)

Description copied from interface: JobMasterGateway

Triggers taking a savepoint of the executed job.

Specified by:

triggerSavepoint in interface JobMasterGateway

Parameters:

targetDirectory - to which to write the savepoint data or null if the default savepoint directory should be used

timeout - for the rpc call

Returns:

Future which is completed with the savepoint path once completed

requestOperatorBackPressureStats

public CompletableFuture<OperatorBackPressureStatsResponse> requestOperatorBackPressureStats(JobVertexID jobVertexId)

Description copied from interface: JobMasterGateway

Requests the statistics on operator back pressure.

Specified by:

requestOperatorBackPressureStats in interface JobMasterGateway

Parameters:

jobVertexId - JobVertex for which the stats are requested.

Returns:

A Future to the OperatorBackPressureStatsResponse or null if the stats are not available (yet).

notifyAllocationFailure

public void notifyAllocationFailure(AllocationID allocationID,
                                    Exception cause)

Description copied from interface: JobMasterGateway

Notifies that the allocation has failed.

Specified by:

notifyAllocationFailure in interface JobMasterGateway

Parameters:

allocationID - the failed allocation id.

cause - the reason that the allocation failed