public class AdaptiveBatchScheduler extends DefaultScheduler
executionDeployer, executionSlotAllocator, log
executionVertexVersioner, inputsLocationsRetriever, jobManagerJobMetricGroup, operatorCoordinatorHandler, stateLocationRetriever
Constructor and Description |
---|
AdaptiveBatchScheduler(org.slf4j.Logger log,
JobGraph jobGraph,
Executor ioExecutor,
Configuration jobMasterConfiguration,
java.util.function.Consumer<ComponentMainThreadExecutor> startUpAction,
ScheduledExecutor delayExecutor,
ClassLoader userCodeLoader,
CheckpointsCleaner checkpointsCleaner,
CheckpointRecoveryFactory checkpointRecoveryFactory,
JobManagerJobMetricGroup jobManagerJobMetricGroup,
SchedulingStrategyFactory schedulingStrategyFactory,
FailoverStrategy.Factory failoverStrategyFactory,
RestartBackoffTimeStrategy restartBackoffTimeStrategy,
ExecutionOperations executionOperations,
ExecutionVertexVersioner executionVertexVersioner,
ExecutionSlotAllocatorFactory executionSlotAllocatorFactory,
long initializationTimestamp,
ComponentMainThreadExecutor mainThreadExecutor,
JobStatusListener jobStatusListener,
ExecutionGraphFactory executionGraphFactory,
ShuffleMaster<?> shuffleMaster,
Time rpcTimeout,
VertexParallelismDecider vertexParallelismDecider,
int defaultMaxParallelism) |
Modifier and Type | Method and Description |
---|---|
static VertexParallelismStore |
computeVertexParallelismStoreForDynamicGraph(Iterable<JobVertex> vertices,
int defaultMaxParallelism)
Compute the
VertexParallelismStore for all given vertices in a dynamic graph, which
will set defaults and ensure that the returned store contains valid parallelisms, with the
configured default max parallelism. |
protected void |
onTaskFinished(Execution execution) |
protected void |
startSchedulingInternal() |
allocateSlotsAndDeploy, cancelAllPendingSlotRequestsForVertex, cancelAllPendingSlotRequestsInternal, cancelExecution, createFailureHandlingResultSnapshot, getNumberOfRestarts, handleGlobalFailure, handleTaskFailure, onTaskFailed, recordTaskFailure
acknowledgeCheckpoint, archiveFromFailureHandlingResult, archiveGlobalFailure, cancel, closeAsync, computeVertexParallelismStore, computeVertexParallelismStore, computeVertexParallelismStore, computeVertexParallelismStore, declineCheckpoint, deliverCoordinationRequestToCoordinator, deliverOperatorEventToCoordinator, failJob, getDefaultMaxParallelism, getExceptionHistory, getExecutionGraph, getExecutionJobVertex, getExecutionVertex, getJobGraph, getJobTerminationFuture, getMainThreadExecutor, getResultPartitionAvailabilityChecker, getSchedulingTopology, notifyKvStateRegistered, notifyKvStateUnregistered, registerJobMetrics, reportCheckpointMetrics, requestJob, requestJobDetails, requestJobStatus, requestKvStateLocation, requestNextInputSplit, requestPartitionState, resetForNewExecution, resetForNewExecutions, restoreState, setGlobalFailureCause, startCheckpointScheduler, startScheduling, stopCheckpointScheduler, stopWithSavepoint, transitionExecutionGraphState, transitionToRunning, transitionToScheduled, triggerCheckpoint, triggerSavepoint, updateAccumulators, updateTaskExecutionState
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
updateTaskExecutionState
close
public AdaptiveBatchScheduler(org.slf4j.Logger log, JobGraph jobGraph, Executor ioExecutor, Configuration jobMasterConfiguration, java.util.function.Consumer<ComponentMainThreadExecutor> startUpAction, ScheduledExecutor delayExecutor, ClassLoader userCodeLoader, CheckpointsCleaner checkpointsCleaner, CheckpointRecoveryFactory checkpointRecoveryFactory, JobManagerJobMetricGroup jobManagerJobMetricGroup, SchedulingStrategyFactory schedulingStrategyFactory, FailoverStrategy.Factory failoverStrategyFactory, RestartBackoffTimeStrategy restartBackoffTimeStrategy, ExecutionOperations executionOperations, ExecutionVertexVersioner executionVertexVersioner, ExecutionSlotAllocatorFactory executionSlotAllocatorFactory, long initializationTimestamp, ComponentMainThreadExecutor mainThreadExecutor, JobStatusListener jobStatusListener, ExecutionGraphFactory executionGraphFactory, ShuffleMaster<?> shuffleMaster, Time rpcTimeout, VertexParallelismDecider vertexParallelismDecider, int defaultMaxParallelism) throws Exception
Exception
protected void startSchedulingInternal()
startSchedulingInternal
in class DefaultScheduler
protected void onTaskFinished(Execution execution)
onTaskFinished
in class DefaultScheduler
@VisibleForTesting public static VertexParallelismStore computeVertexParallelismStoreForDynamicGraph(Iterable<JobVertex> vertices, int defaultMaxParallelism)
VertexParallelismStore
for all given vertices in a dynamic graph, which
will set defaults and ensure that the returned store contains valid parallelisms, with the
configured default max parallelism.vertices
- the vertices to compute parallelism fordefaultMaxParallelism
- the global default max parallelismCopyright © 2014–2024 The Apache Software Foundation. All rights reserved.