org.apache.flink.runtime.scheduler

Class SchedulerBase

java.lang.Object

org.apache.flink.runtime.scheduler.SchedulerBase

All Implemented Interfaces:

AutoCloseable, CheckpointScheduling, GlobalFailureHandler, SchedulerNG, AutoCloseableAsync

Direct Known Subclasses:

DefaultScheduler

public abstract class SchedulerBase
extends Object
implements SchedulerNG, CheckpointScheduling

Base class which can be used to implement SchedulerNG.

Field Summary

Fields

Modifier and Type	Field and Description
protected ExecutionVertexVersioner	executionVertexVersioner
protected InputsLocationsRetriever	inputsLocationsRetriever
protected JobManagerJobMetricGroup	jobManagerJobMetricGroup
protected OperatorCoordinatorHandler	operatorCoordinatorHandler
protected StateLocationRetriever	stateLocationRetriever

Constructor Summary

Constructors

Constructor and Description

SchedulerBase(org.slf4j.Logger log, JobGraph jobGraph, Executor ioExecutor, Configuration jobMasterConfiguration, CheckpointsCleaner checkpointsCleaner, CheckpointRecoveryFactory checkpointRecoveryFactory, JobManagerJobMetricGroup jobManagerJobMetricGroup, ExecutionVertexVersioner executionVertexVersioner, long initializationTimestamp, ComponentMainThreadExecutor mainThreadExecutor, JobStatusListener jobStatusListener, ExecutionGraphFactory executionGraphFactory, VertexParallelismStore vertexParallelismStore)

Method Summary

Modifier and Type	Method and Description
void	acknowledgeCheckpoint(JobID jobID, ExecutionAttemptID executionAttemptID, long checkpointId, CheckpointMetrics checkpointMetrics, TaskStateSnapshot checkpointState)
protected void	archiveFromFailureHandlingResult(FailureHandlingResultSnapshot failureHandlingResult)
protected void	archiveGlobalFailure(Throwable failure)
void	cancel()
protected abstract void	cancelAllPendingSlotRequestsInternal()
CompletableFuture<Void>	closeAsync() Trigger the closing of the resource and return the corresponding close future.
static VertexParallelismStore	computeVertexParallelismStore(Iterable<JobVertex> vertices) Compute the VertexParallelismStore for all given vertices, which will set defaults and ensure that the returned store contains valid parallelisms.
static VertexParallelismStore	computeVertexParallelismStore(Iterable<JobVertex> vertices, java.util.function.Function<JobVertex,Integer> defaultMaxParallelismFunc)
static VertexParallelismStore	computeVertexParallelismStore(Iterable<JobVertex> vertices, java.util.function.Function<JobVertex,Integer> defaultMaxParallelismFunc, java.util.function.Function<Integer,Integer> normalizeParallelismFunc) Compute the VertexParallelismStore for all given vertices, which will set defaults and ensure that the returned store contains valid parallelisms, with a custom function for default max parallelism calculation and a custom function for normalizing vertex parallelism.
static VertexParallelismStore	computeVertexParallelismStore(JobGraph jobGraph) Compute the VertexParallelismStore for all vertices of a given job graph, which will set defaults and ensure that the returned store contains valid parallelisms.
void	declineCheckpoint(DeclineCheckpoint decline)
CompletableFuture<CoordinationResponse>	deliverCoordinationRequestToCoordinator(OperatorID operator, CoordinationRequest request) Delivers a coordination request to the OperatorCoordinator with the given OperatorID and returns the coordinator's response.
void	deliverOperatorEventToCoordinator(ExecutionAttemptID taskExecutionId, OperatorID operatorId, OperatorEvent evt) Delivers the given OperatorEvent to the OperatorCoordinator with the given OperatorID.
protected void	failJob(Throwable cause, long timestamp)
static int	getDefaultMaxParallelism(JobVertex vertex) Get a default value to use for a given vertex's max parallelism if none was specified.
Iterable<RootExceptionHistoryEntry>	getExceptionHistory()
ExecutionGraph	getExecutionGraph() ExecutionGraph is exposed to make it easier to rework tests to be based on the new scheduler.
ExecutionJobVertex	getExecutionJobVertex(JobVertexID jobVertexId)
ExecutionVertex	getExecutionVertex(ExecutionVertexID executionVertexId)
protected JobGraph	getJobGraph()
CompletableFuture<JobStatus>	getJobTerminationFuture()
protected ComponentMainThreadExecutor	getMainThreadExecutor()
protected abstract long	getNumberOfRestarts()
protected ResultPartitionAvailabilityChecker	getResultPartitionAvailabilityChecker()
protected SchedulingTopology	getSchedulingTopology()
void	notifyKvStateRegistered(JobID jobId, JobVertexID jobVertexId, KeyGroupRange keyGroupRange, String registrationName, KvStateID kvStateId, InetSocketAddress kvStateServerAddress)
void	notifyKvStateUnregistered(JobID jobId, JobVertexID jobVertexId, KeyGroupRange keyGroupRange, String registrationName)
protected abstract void	onTaskFailed(Execution execution)
protected abstract void	onTaskFinished(Execution execution)
static void	registerJobMetrics(MetricGroup metrics, JobStatusProvider jobStatusProvider, Gauge<Long> numberOfRestarts, DeploymentStateTimeMetrics deploymentTimeMetrics, java.util.function.Consumer<JobStatusListener> jobStatusListenerRegistrar, long initializationTimestamp, MetricOptions.JobStatusMetricsSettings jobStatusMetricsSettings)
void	reportCheckpointMetrics(JobID jobID, ExecutionAttemptID attemptId, long id, CheckpointMetrics metrics)
ExecutionGraphInfo	requestJob()
JobDetails	requestJobDetails()
JobStatus	requestJobStatus()
KvStateLocation	requestKvStateLocation(JobID jobId, String registrationName)
SerializedInputSplit	requestNextInputSplit(JobVertexID vertexID, ExecutionAttemptID executionAttempt)
ExecutionState	requestPartitionState(IntermediateDataSetID intermediateResultId, ResultPartitionID resultPartitionId)
protected void	resetForNewExecution(ExecutionVertexID executionVertexId)
protected void	resetForNewExecutions(Collection<ExecutionVertexID> vertices)
protected void	restoreState(Set<ExecutionVertexID> vertices, boolean isGlobalRecovery)
protected void	setGlobalFailureCause(Throwable cause, long timestamp)
void	startCheckpointScheduler() Starts the periodic scheduling if possible.
void	startScheduling()
protected abstract void	startSchedulingInternal()
void	stopCheckpointScheduler() Stops the periodic scheduling if possible.
CompletableFuture<String>	stopWithSavepoint(String targetDirectory, boolean terminate, SavepointFormatType formatType)
protected void	transitionExecutionGraphState(JobStatus current, JobStatus newState)
protected void	transitionToRunning()
protected void	transitionToScheduled(List<ExecutionVertexID> verticesToDeploy)
CompletableFuture<String>	triggerCheckpoint()
CompletableFuture<String>	triggerSavepoint(String targetDirectory, boolean cancelJob, SavepointFormatType formatType)
void	updateAccumulators(AccumulatorSnapshot accumulatorSnapshot)
boolean	updateTaskExecutionState(TaskExecutionStateTransition taskExecutionState)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.scheduler.SchedulerNG

updateTaskExecutionState

Methods inherited from interface org.apache.flink.runtime.scheduler.GlobalFailureHandler

handleGlobalFailure

Methods inherited from interface org.apache.flink.util.AutoCloseableAsync

close

Field Detail

stateLocationRetriever

protected final StateLocationRetriever stateLocationRetriever

inputsLocationsRetriever

protected final InputsLocationsRetriever inputsLocationsRetriever

jobManagerJobMetricGroup

protected final JobManagerJobMetricGroup jobManagerJobMetricGroup

executionVertexVersioner

protected final ExecutionVertexVersioner executionVertexVersioner

operatorCoordinatorHandler

protected final OperatorCoordinatorHandler operatorCoordinatorHandler

Constructor Detail

SchedulerBase

public SchedulerBase(org.slf4j.Logger log,
                     JobGraph jobGraph,
                     Executor ioExecutor,
                     Configuration jobMasterConfiguration,
                     CheckpointsCleaner checkpointsCleaner,
                     CheckpointRecoveryFactory checkpointRecoveryFactory,
                     JobManagerJobMetricGroup jobManagerJobMetricGroup,
                     ExecutionVertexVersioner executionVertexVersioner,
                     long initializationTimestamp,
                     ComponentMainThreadExecutor mainThreadExecutor,
                     JobStatusListener jobStatusListener,
                     ExecutionGraphFactory executionGraphFactory,
                     VertexParallelismStore vertexParallelismStore)
              throws Exception

Throws:

Exception

Method Detail

getDefaultMaxParallelism

public static int getDefaultMaxParallelism(JobVertex vertex)

Get a default value to use for a given vertex's max parallelism if none was specified.

Parameters:

vertex - the vertex to compute a default max parallelism for

Returns:

the computed max parallelism

computeVertexParallelismStore

public static VertexParallelismStore computeVertexParallelismStore(Iterable<JobVertex> vertices,
                                                                   java.util.function.Function<JobVertex,Integer> defaultMaxParallelismFunc)

computeVertexParallelismStore

public static VertexParallelismStore computeVertexParallelismStore(Iterable<JobVertex> vertices,
                                                                   java.util.function.Function<JobVertex,Integer> defaultMaxParallelismFunc,
                                                                   java.util.function.Function<Integer,Integer> normalizeParallelismFunc)

Compute the VertexParallelismStore for all given vertices, which will set defaults and ensure that the returned store contains valid parallelisms, with a custom function for default max parallelism calculation and a custom function for normalizing vertex parallelism.

Parameters:

vertices - the vertices to compute parallelism for

defaultMaxParallelismFunc - a function for computing a default max parallelism if none is specified on a given vertex

normalizeParallelismFunc - a function for normalizing vertex parallelism

Returns:

the computed parallelism store

computeVertexParallelismStore

public static VertexParallelismStore computeVertexParallelismStore(Iterable<JobVertex> vertices)

Compute the VertexParallelismStore for all given vertices, which will set defaults and ensure that the returned store contains valid parallelisms.

Parameters:

vertices - the vertices to compute parallelism for

Returns:

the computed parallelism store

computeVertexParallelismStore

public static VertexParallelismStore computeVertexParallelismStore(JobGraph jobGraph)

Compute the VertexParallelismStore for all vertices of a given job graph, which will set defaults and ensure that the returned store contains valid parallelisms.

Parameters:

jobGraph - the job graph to retrieve vertices from

Returns:

the computed parallelism store

resetForNewExecutions

protected void resetForNewExecutions(Collection<ExecutionVertexID> vertices)

resetForNewExecution

protected void resetForNewExecution(ExecutionVertexID executionVertexId)

restoreState

protected void restoreState(Set<ExecutionVertexID> vertices,
                            boolean isGlobalRecovery)
                     throws Exception

Throws:

Exception

transitionToScheduled

protected void transitionToScheduled(List<ExecutionVertexID> verticesToDeploy)

setGlobalFailureCause

protected void setGlobalFailureCause(@Nullable
                                     Throwable cause,
                                     long timestamp)

getMainThreadExecutor

protected ComponentMainThreadExecutor getMainThreadExecutor()

failJob

protected void failJob(Throwable cause,
                       long timestamp)

getSchedulingTopology

protected final SchedulingTopology getSchedulingTopology()

getResultPartitionAvailabilityChecker

protected final ResultPartitionAvailabilityChecker getResultPartitionAvailabilityChecker()

transitionToRunning

protected final void transitionToRunning()

getExecutionVertex

public ExecutionVertex getExecutionVertex(ExecutionVertexID executionVertexId)

getExecutionJobVertex

public ExecutionJobVertex getExecutionJobVertex(JobVertexID jobVertexId)

getJobGraph

protected JobGraph getJobGraph()

getNumberOfRestarts

protected abstract long getNumberOfRestarts()

cancelAllPendingSlotRequestsInternal

protected abstract void cancelAllPendingSlotRequestsInternal()

transitionExecutionGraphState

protected void transitionExecutionGraphState(JobStatus current,
                                             JobStatus newState)

getExecutionGraph

@VisibleForTesting
public ExecutionGraph getExecutionGraph()

ExecutionGraph is exposed to make it easier to rework tests to be based on the new scheduler. ExecutionGraph is expected to be used only for state check. Yet at the moment, before all the actions are factored out from ExecutionGraph and its sub-components, some actions may still be performed directly on it.

startScheduling

public final void startScheduling()

Specified by:

startScheduling in interface SchedulerNG

registerJobMetrics

public static void registerJobMetrics(MetricGroup metrics,
                                      JobStatusProvider jobStatusProvider,
                                      Gauge<Long> numberOfRestarts,
                                      DeploymentStateTimeMetrics deploymentTimeMetrics,
                                      java.util.function.Consumer<JobStatusListener> jobStatusListenerRegistrar,
                                      long initializationTimestamp,
                                      MetricOptions.JobStatusMetricsSettings jobStatusMetricsSettings)

startSchedulingInternal

protected abstract void startSchedulingInternal()

closeAsync

public CompletableFuture<Void> closeAsync()

Description copied from interface: AutoCloseableAsync

Trigger the closing of the resource and return the corresponding close future.

Specified by:

closeAsync in interface AutoCloseableAsync

Returns:

Future which is completed once the resource has been closed

cancel

public void cancel()

Specified by:

cancel in interface SchedulerNG

getJobTerminationFuture

public CompletableFuture<JobStatus> getJobTerminationFuture()

Specified by:

getJobTerminationFuture in interface SchedulerNG

archiveGlobalFailure

protected final void archiveGlobalFailure(Throwable failure)

archiveFromFailureHandlingResult

protected final void archiveFromFailureHandlingResult(FailureHandlingResultSnapshot failureHandlingResult)

updateTaskExecutionState

public final boolean updateTaskExecutionState(TaskExecutionStateTransition taskExecutionState)

Specified by:

updateTaskExecutionState in interface SchedulerNG

onTaskFinished

protected abstract void onTaskFinished(Execution execution)

onTaskFailed

protected abstract void onTaskFailed(Execution execution)

requestNextInputSplit

public SerializedInputSplit requestNextInputSplit(JobVertexID vertexID,
                                                  ExecutionAttemptID executionAttempt)
                                           throws IOException

Specified by:

requestNextInputSplit in interface SchedulerNG

Throws:

IOException

requestPartitionState

public ExecutionState requestPartitionState(IntermediateDataSetID intermediateResultId,
                                            ResultPartitionID resultPartitionId)
                                     throws PartitionProducerDisposedException

Specified by:

requestPartitionState in interface SchedulerNG

Throws:

PartitionProducerDisposedException

getExceptionHistory

@VisibleForTesting
public Iterable<RootExceptionHistoryEntry> getExceptionHistory()

requestJob

public ExecutionGraphInfo requestJob()

Specified by:

requestJob in interface SchedulerNG

requestJobStatus

public JobStatus requestJobStatus()

Specified by:

requestJobStatus in interface SchedulerNG

requestJobDetails

public JobDetails requestJobDetails()

Specified by:

requestJobDetails in interface SchedulerNG

requestKvStateLocation

public KvStateLocation requestKvStateLocation(JobID jobId,
                                              String registrationName)
                                       throws UnknownKvStateLocation,
                                              FlinkJobNotFoundException

Specified by:

requestKvStateLocation in interface SchedulerNG

Throws:

UnknownKvStateLocation

FlinkJobNotFoundException

notifyKvStateRegistered

public void notifyKvStateRegistered(JobID jobId,
                                    JobVertexID jobVertexId,
                                    KeyGroupRange keyGroupRange,
                                    String registrationName,
                                    KvStateID kvStateId,
                                    InetSocketAddress kvStateServerAddress)
                             throws FlinkJobNotFoundException

Specified by:

notifyKvStateRegistered in interface SchedulerNG

Throws:

FlinkJobNotFoundException

notifyKvStateUnregistered

public void notifyKvStateUnregistered(JobID jobId,
                                      JobVertexID jobVertexId,
                                      KeyGroupRange keyGroupRange,
                                      String registrationName)
                               throws FlinkJobNotFoundException

Specified by:

notifyKvStateUnregistered in interface SchedulerNG

Throws:

FlinkJobNotFoundException

updateAccumulators

public void updateAccumulators(AccumulatorSnapshot accumulatorSnapshot)

Specified by:

updateAccumulators in interface SchedulerNG

triggerSavepoint

public CompletableFuture<String> triggerSavepoint(String targetDirectory,
                                                  boolean cancelJob,
                                                  SavepointFormatType formatType)

Specified by:

triggerSavepoint in interface SchedulerNG

triggerCheckpoint

public CompletableFuture<String> triggerCheckpoint()

Specified by:

triggerCheckpoint in interface SchedulerNG

stopCheckpointScheduler

public void stopCheckpointScheduler()

Description copied from interface: CheckpointScheduling

Stops the periodic scheduling if possible.

Specified by:

stopCheckpointScheduler in interface CheckpointScheduling

startCheckpointScheduler

public void startCheckpointScheduler()

Description copied from interface: CheckpointScheduling

Starts the periodic scheduling if possible.

Specified by:

startCheckpointScheduler in interface CheckpointScheduling

acknowledgeCheckpoint

public void acknowledgeCheckpoint(JobID jobID,
                                  ExecutionAttemptID executionAttemptID,
                                  long checkpointId,
                                  CheckpointMetrics checkpointMetrics,
                                  TaskStateSnapshot checkpointState)

Specified by:

acknowledgeCheckpoint in interface SchedulerNG

declineCheckpoint

public void declineCheckpoint(DeclineCheckpoint decline)

Specified by:

declineCheckpoint in interface SchedulerNG

reportCheckpointMetrics

public void reportCheckpointMetrics(JobID jobID,
                                    ExecutionAttemptID attemptId,
                                    long id,
                                    CheckpointMetrics metrics)

Specified by:

reportCheckpointMetrics in interface SchedulerNG

stopWithSavepoint

public CompletableFuture<String> stopWithSavepoint(@Nullable
                                                   String targetDirectory,
                                                   boolean terminate,
                                                   SavepointFormatType formatType)

Specified by:

stopWithSavepoint in interface SchedulerNG

deliverOperatorEventToCoordinator

public void deliverOperatorEventToCoordinator(ExecutionAttemptID taskExecutionId,
                                              OperatorID operatorId,
                                              OperatorEvent evt)
                                       throws FlinkException

Description copied from interface: SchedulerNG

Delivers the given OperatorEvent to the OperatorCoordinator with the given OperatorID.

Failure semantics: If the task manager sends an event for a non-running task or a non-existing operator coordinator, then respond with an exception to the call. If task and coordinator exist, then we assume that the call from the TaskManager was valid, and any bubbling exception needs to cause a job failure

Specified by:

deliverOperatorEventToCoordinator in interface SchedulerNG

Throws:

FlinkException - Thrown, if the task is not running or no operator/coordinator exists for the given ID.

deliverCoordinationRequestToCoordinator

public CompletableFuture<CoordinationResponse> deliverCoordinationRequestToCoordinator(OperatorID operator,
                                                                                       CoordinationRequest request)
                                                                                throws FlinkException

Description copied from interface: SchedulerNG

Delivers a coordination request to the OperatorCoordinator with the given OperatorID and returns the coordinator's response.

Specified by:

deliverCoordinationRequestToCoordinator in interface SchedulerNG

Returns:

A future containing the response.

Throws:

FlinkException - Thrown, if the task is not running, or no operator/coordinator exists for the given ID, or the coordinator cannot handle client events.