org.apache.flink.runtime.scheduler.adaptivebatch

Class SpeculativeScheduler

java.lang.Object

org.apache.flink.runtime.scheduler.SchedulerBase

org.apache.flink.runtime.scheduler.DefaultScheduler

org.apache.flink.runtime.scheduler.adaptivebatch.AdaptiveBatchScheduler

org.apache.flink.runtime.scheduler.adaptivebatch.SpeculativeScheduler

All Implemented Interfaces:

AutoCloseable, CheckpointScheduling, GlobalFailureHandler, SchedulerNG, SchedulerOperations, SlowTaskDetectorListener, AutoCloseableAsync

public class SpeculativeScheduler
extends AdaptiveBatchScheduler
implements SlowTaskDetectorListener

The speculative scheduler.

Field Summary

Fields inherited from class org.apache.flink.runtime.scheduler.DefaultScheduler

executionDeployer, executionSlotAllocator, log

Fields inherited from class org.apache.flink.runtime.scheduler.SchedulerBase

executionVertexVersioner, inputsLocationsRetriever, jobInfo, jobManagerJobMetricGroup, operatorCoordinatorHandler, stateLocationRetriever

Constructor Summary

Constructors

Constructor and Description

SpeculativeScheduler(org.slf4j.Logger log, JobGraph jobGraph, Executor ioExecutor, Configuration jobMasterConfiguration, Consumer<ComponentMainThreadExecutor> startUpAction, ScheduledExecutor delayExecutor, ClassLoader userCodeLoader, CheckpointsCleaner checkpointsCleaner, CheckpointRecoveryFactory checkpointRecoveryFactory, JobManagerJobMetricGroup jobManagerJobMetricGroup, SchedulingStrategyFactory schedulingStrategyFactory, FailoverStrategy.Factory failoverStrategyFactory, RestartBackoffTimeStrategy restartBackoffTimeStrategy, ExecutionOperations executionOperations, ExecutionVertexVersioner executionVertexVersioner, ExecutionSlotAllocatorFactory executionSlotAllocatorFactory, long initializationTimestamp, ComponentMainThreadExecutor mainThreadExecutor, JobStatusListener jobStatusListener, Collection<FailureEnricher> failureEnrichers, ExecutionGraphFactory executionGraphFactory, ShuffleMaster<?> shuffleMaster, Time rpcTimeout, VertexParallelismAndInputInfosDecider vertexParallelismAndInputInfosDecider, int defaultMaxParallelism, BlocklistOperations blocklistOperations, JobManagerOptions.HybridPartitionDataConsumeConstraint hybridPartitionDataConsumeConstraint, Map<JobVertexID,ForwardGroup> forwardGroupsByJobVertexId)

Method Summary

Modifier and Type	Method and Description
CompletableFuture<Void>	closeAsync() Trigger the closing of the resource and return the corresponding close future.
SpeculativeExecutionVertex	getExecutionVertex(ExecutionVertexID executionVertexId)
protected void	handleTaskFailure(Execution failedExecution, Throwable error)
void	notifySlowTasks(Map<ExecutionVertexID,Collection<ExecutionAttemptID>> slowTasks) Notify detected slow tasks.
protected void	onTaskFailed(Execution execution)
protected void	onTaskFinished(Execution execution, IOMetrics ioMetrics)
protected void	resetForNewExecution(ExecutionVertexID executionVertexId)
protected void	startSchedulingInternal()

Methods inherited from class org.apache.flink.runtime.scheduler.adaptivebatch.AdaptiveBatchScheduler

allocateSlotsAndDeploy, computeDynamicSourceParallelism, computeVertexParallelismStoreForDynamicGraph, getMarkPartitionFinishedStrategy, initializeVerticesIfPossible

Methods inherited from class org.apache.flink.runtime.scheduler.DefaultScheduler

cancelAllPendingSlotRequestsForVertex, cancelAllPendingSlotRequestsInternal, cancelExecution, createFailureHandlingResultSnapshot, getNumberOfRestarts, handleGlobalFailure, recordTaskFailure

Methods inherited from class org.apache.flink.runtime.scheduler.SchedulerBase

acknowledgeCheckpoint, archiveFromFailureHandlingResult, archiveGlobalFailure, cancel, computeVertexParallelismStore, computeVertexParallelismStore, computeVertexParallelismStore, computeVertexParallelismStore, declineCheckpoint, deliverCoordinationRequestToCoordinator, deliverOperatorEventToCoordinator, failJob, getDefaultMaxParallelism, getExceptionHistory, getExecutionGraph, getExecutionJobVertex, getJobGraph, getJobTerminationFuture, getMainThreadExecutor, getResultPartitionAvailabilityChecker, getSchedulingTopology, notifyEndOfData, notifyKvStateRegistered, notifyKvStateUnregistered, registerJobMetrics, reportCheckpointMetrics, reportInitializationMetrics, requestCheckpointStats, requestJob, requestJobStatus, requestKvStateLocation, requestNextInputSplit, requestPartitionState, resetForNewExecutions, restoreState, setGlobalFailureCause, startCheckpointScheduler, startScheduling, stopCheckpointScheduler, stopWithSavepoint, transitionExecutionGraphState, transitionToRunning, transitionToScheduled, triggerCheckpoint, triggerSavepoint, updateAccumulators, updateTaskExecutionState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.scheduler.SchedulerNG

requestJobResourceRequirements, updateJobResourceRequirements, updateTaskExecutionState

Methods inherited from interface org.apache.flink.util.AutoCloseableAsync

close

Constructor Detail

SpeculativeScheduler

public SpeculativeScheduler(org.slf4j.Logger log,
                            JobGraph jobGraph,
                            Executor ioExecutor,
                            Configuration jobMasterConfiguration,
                            Consumer<ComponentMainThreadExecutor> startUpAction,
                            ScheduledExecutor delayExecutor,
                            ClassLoader userCodeLoader,
                            CheckpointsCleaner checkpointsCleaner,
                            CheckpointRecoveryFactory checkpointRecoveryFactory,
                            JobManagerJobMetricGroup jobManagerJobMetricGroup,
                            SchedulingStrategyFactory schedulingStrategyFactory,
                            FailoverStrategy.Factory failoverStrategyFactory,
                            RestartBackoffTimeStrategy restartBackoffTimeStrategy,
                            ExecutionOperations executionOperations,
                            ExecutionVertexVersioner executionVertexVersioner,
                            ExecutionSlotAllocatorFactory executionSlotAllocatorFactory,
                            long initializationTimestamp,
                            ComponentMainThreadExecutor mainThreadExecutor,
                            JobStatusListener jobStatusListener,
                            Collection<FailureEnricher> failureEnrichers,
                            ExecutionGraphFactory executionGraphFactory,
                            ShuffleMaster<?> shuffleMaster,
                            Time rpcTimeout,
                            VertexParallelismAndInputInfosDecider vertexParallelismAndInputInfosDecider,
                            int defaultMaxParallelism,
                            BlocklistOperations blocklistOperations,
                            JobManagerOptions.HybridPartitionDataConsumeConstraint hybridPartitionDataConsumeConstraint,
                            Map<JobVertexID,ForwardGroup> forwardGroupsByJobVertexId)
                     throws Exception

Throws:

Exception

Method Detail

startSchedulingInternal

protected void startSchedulingInternal()

Overrides:

startSchedulingInternal in class AdaptiveBatchScheduler

closeAsync

public CompletableFuture<Void> closeAsync()

Description copied from interface: AutoCloseableAsync

Trigger the closing of the resource and return the corresponding close future.

Specified by:

closeAsync in interface AutoCloseableAsync

Overrides:

closeAsync in class SchedulerBase

Returns:

Future which is completed once the resource has been closed

getExecutionVertex

public SpeculativeExecutionVertex getExecutionVertex(ExecutionVertexID executionVertexId)

Overrides:

getExecutionVertex in class SchedulerBase

onTaskFinished

protected void onTaskFinished(Execution execution,
                              IOMetrics ioMetrics)

Overrides:

onTaskFinished in class AdaptiveBatchScheduler

onTaskFailed

protected void onTaskFailed(Execution execution)

Overrides:

onTaskFailed in class DefaultScheduler

handleTaskFailure

protected void handleTaskFailure(Execution failedExecution,
                                 @Nullable
                                 Throwable error)

Overrides:

handleTaskFailure in class DefaultScheduler

resetForNewExecution

protected void resetForNewExecution(ExecutionVertexID executionVertexId)

Overrides:

resetForNewExecution in class AdaptiveBatchScheduler

notifySlowTasks

public void notifySlowTasks(Map<ExecutionVertexID,Collection<ExecutionAttemptID>> slowTasks)

Description copied from interface: SlowTaskDetectorListener

Notify detected slow tasks.

Specified by:

notifySlowTasks in interface SlowTaskDetectorListener

Parameters:

slowTasks - the map of execution vertices and their execution attempts which are detected as slow.