org.apache.flink.runtime.executiongraph

Class ExecutionGraph

java.lang.Object

org.apache.flink.runtime.executiongraph.ExecutionGraph

All Implemented Interfaces:

AccessExecutionGraph

public class ExecutionGraph
extends Object
implements AccessExecutionGraph

The execution graph is the central data structure that coordinates the distributed execution of a data flow. It keeps representations of each parallel task, each intermediate stream, and the communication between them.

The execution graph consists of the following constructs:

• The ExecutionJobVertex represents one vertex from the JobGraph (usually one operation like "map" or "join") during execution. It holds the aggregated state of all parallel subtasks. The ExecutionJobVertex is identified inside the graph by the JobVertexID, which it takes from the JobGraph's corresponding JobVertex.
• The ExecutionVertex represents one parallel subtask. For each ExecutionJobVertex, there are as many ExecutionVertices as the parallelism. The ExecutionVertex is identified by the ExecutionJobVertex and the index of the parallel subtask
• The Execution is one attempt to execute a ExecutionVertex. There may be multiple Executions for the ExecutionVertex, in case of a failure, or in the case where some data needs to be recomputed because it is no longer available when requested by later operations. An Execution is always identified by an ExecutionAttemptID. All messages between the JobManager and the TaskManager about deployment of tasks and updates in the task status always use the ExecutionAttemptID to address the message receiver.

Global and local failover

The Execution Graph has two failover modes: global failover and local failover.

A global failover aborts the task executions for all vertices and restarts whole data flow graph from the last completed checkpoint. Global failover is considered the "fallback strategy" that is used when a local failover is unsuccessful, or when a issue is found in the state of the ExecutionGraph that could mark it as inconsistent (caused by a bug).

A local failover is triggered when an individual vertex execution (a task) fails. The local failover is coordinated by the FailoverStrategy. A local failover typically attempts to restart as little as possible, but as much as necessary.

Between local- and global failover, the global failover always takes precedence, because it is the core mechanism that the ExecutionGraph relies on to bring back consistency. The guard that, the ExecutionGraph maintains a global modification version, which is incremented with every global failover (and other global actions, like job cancellation, or terminal failure). Local failover is always scoped by the modification version that the execution graph had when the failover was triggered. If a new global modification version is reached during local failover (meaning there is a concurrent global failover), the failover strategy has to yield before the global failover.

Constructor Summary

Constructors

Constructor and Description

ExecutionGraph(JobInformation jobInformation, ScheduledExecutorService futureExecutor, Executor ioExecutor, Time rpcTimeout, RestartStrategy restartStrategy, int maxPriorAttemptsHistoryLength, FailoverStrategy.Factory failoverStrategyFactory, SlotProvider slotProvider, ClassLoader userClassLoader, BlobWriter blobWriter, Time allocationTimeout, PartitionReleaseStrategy.Factory partitionReleaseStrategyFactory, ShuffleMaster<?> shuffleMaster, JobMasterPartitionTracker partitionTracker, ScheduleMode scheduleMode)

Method Summary

Modifier and Type	Method and Description
Map<String,OptionalFailure<Accumulator<?,?>>>	aggregateUserAccumulators() Merges all accumulator results from the tasks previously executed in the Executions.
void	attachJobGraph(List<JobVertex> topologiallySorted)
void	cancel()
void	enableCheckpointing(CheckpointCoordinatorConfiguration chkConfig, List<ExecutionJobVertex> verticesToTrigger, List<ExecutionJobVertex> verticesToWaitFor, List<ExecutionJobVertex> verticesToCommitTo, List<MasterTriggerRestoreHook<?>> masterHooks, CheckpointIDCounter checkpointIDCounter, CompletedCheckpointStore checkpointStore, StateBackend checkpointStateBackend, CheckpointStatsTracker statsTracker)
void	enableNgScheduling(InternalFailuresListener internalTaskFailuresListener)
void	failGlobal(Throwable t) Fails the execution graph globally.
void	failJob(Throwable cause)
StringifiedAccumulatorResult[]	getAccumulatorResultsStringified() Returns the a stringified version of the user-defined accumulators.
Map<String,SerializedValue<OptionalFailure<Object>>>	getAccumulatorsSerialized() Gets a serialized accumulator map.
Iterable<ExecutionVertex>	getAllExecutionVertices() Returns an iterable containing all execution vertices for this execution graph.
Map<IntermediateDataSetID,IntermediateResult>	getAllIntermediateResults()
Time	getAllocationTimeout()
Map<JobVertexID,ExecutionJobVertex>	getAllVertices() Returns a map containing all job vertices for this execution graph.
ArchivedExecutionConfig	getArchivedExecutionConfig() Returns the serializable ArchivedExecutionConfig.
BlobWriter	getBlobWriter()
CheckpointCoordinator	getCheckpointCoordinator()
CheckpointCoordinatorConfiguration	getCheckpointCoordinatorConfiguration() Returns the CheckpointCoordinatorConfiguration or null if checkpointing is disabled.
CheckpointStatsSnapshot	getCheckpointStatsSnapshot() Returns a snapshot of the checkpoint statistics or null if checkpointing is disabled.
FailoverStrategy	getFailoverStrategy() Gets the failover strategy used by the execution graph to recover from failures of tasks.
FailoverTopology<?,?>	getFailoverTopology()
Throwable	getFailureCause()
ErrorInfo	getFailureInfo() Returns the exception that caused the job to fail.
Executor	getFutureExecutor() Returns the ExecutionContext associated with this ExecutionGraph.
long	getGlobalModVersion() Gets the current global modification version of the ExecutionGraph.
Configuration	getJobConfiguration()
JobID	getJobID() Returns the JobID for this execution graph.
Either<SerializedValue<JobInformation>,PermanentBlobKey>	getJobInformationOrBlobKey()
ComponentMainThreadExecutor	getJobMasterMainThreadExecutor()
String	getJobName() Returns the job name for the execution graph.
ExecutionJobVertex	getJobVertex(JobVertexID id) Returns the job vertex for the given JobVertexID.
String	getJsonPlan() Returns the job plan as a JSON string.
KvStateLocationRegistry	getKvStateLocationRegistry()
int	getNumberOfExecutionJobVertices() Gets the number of job vertices currently held by this execution graph.
long	getNumberOfRestarts() Gets the number of restarts, including full restarts and fine grained restarts.
JobMasterPartitionTracker	getPartitionTracker()
Map<ExecutionAttemptID,Execution>	getRegisteredExecutions()
RestartStrategy	getRestartStrategy()
ResultPartitionAvailabilityChecker	getResultPartitionAvailabilityChecker()
ScheduleMode	getScheduleMode()
SchedulingTopology<?,?>	getSchedulingTopology()
SlotProviderStrategy	getSlotProviderStrategy()
JobStatus	getState() Returns the current JobStatus for this execution graph.
Optional<String>	getStateBackendName() Returns the state backend name for this ExecutionGraph.
long	getStatusTimestamp(JobStatus status) Returns the timestamp for the given JobStatus.
CompletableFuture<JobStatus>	getTerminationFuture() Returns the termination future of this ExecutionGraph.
int	getTotalNumberOfVertices()
ClassLoader	getUserClassLoader()
Iterable<ExecutionJobVertex>	getVerticesTopologically() Returns an iterable containing all job vertices for this execution graph in the order they were created.
void	incrementRestarts()
void	initFailureCause(Throwable t)
boolean	isArchived() Returns whether this execution graph was archived.
boolean	isLegacyScheduling()
boolean	isStoppable() Returns whether the job for this execution graph is stoppable.
void	registerJobStatusListener(JobStatusListener listener)
void	restart(long expectedGlobalVersion)
void	scheduleForExecution()
void	scheduleOrUpdateConsumers(ResultPartitionID partitionId) Schedule or updates consumers of the given result partition.
void	setJsonPlan(String jsonPlan)
void	start(ComponentMainThreadExecutor jobMasterMainThreadExecutor)
void	suspend(Throwable suspensionCause) Suspends the current ExecutionGraph.
boolean	transitionState(JobStatus current, JobStatus newState)
void	transitionToRunning()
void	updateAccumulators(AccumulatorSnapshot accumulatorSnapshot) Updates the accumulators during the runtime of a job.
boolean	updateState(TaskExecutionState state) Updates the state of one of the ExecutionVertex's Execution attempts.
JobStatus	waitUntilTerminal()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ExecutionGraph

public ExecutionGraph(JobInformation jobInformation,
                      ScheduledExecutorService futureExecutor,
                      Executor ioExecutor,
                      Time rpcTimeout,
                      RestartStrategy restartStrategy,
                      int maxPriorAttemptsHistoryLength,
                      FailoverStrategy.Factory failoverStrategyFactory,
                      SlotProvider slotProvider,
                      ClassLoader userClassLoader,
                      BlobWriter blobWriter,
                      Time allocationTimeout,
                      PartitionReleaseStrategy.Factory partitionReleaseStrategyFactory,
                      ShuffleMaster<?> shuffleMaster,
                      JobMasterPartitionTracker partitionTracker,
                      ScheduleMode scheduleMode)
               throws IOException

Throws:

IOException

Method Detail

start

public void start(@Nonnull
                  ComponentMainThreadExecutor jobMasterMainThreadExecutor)

getNumberOfExecutionJobVertices

public int getNumberOfExecutionJobVertices()

Gets the number of job vertices currently held by this execution graph.

Returns:

The current number of job vertices.

getSchedulingTopology

public SchedulingTopology<?,?> getSchedulingTopology()

getFailoverTopology

public FailoverTopology<?,?> getFailoverTopology()

getScheduleMode

public ScheduleMode getScheduleMode()

getAllocationTimeout

public Time getAllocationTimeout()

getJobMasterMainThreadExecutor

@Nonnull
public ComponentMainThreadExecutor getJobMasterMainThreadExecutor()

isArchived

public boolean isArchived()

Description copied from interface: AccessExecutionGraph

Returns whether this execution graph was archived.

Specified by:

isArchived in interface AccessExecutionGraph

Returns:

true, if the execution graph was archived, false otherwise

getStateBackendName

public Optional<String> getStateBackendName()

Description copied from interface: AccessExecutionGraph

Returns the state backend name for this ExecutionGraph.

Specified by:

getStateBackendName in interface AccessExecutionGraph

Returns:

The state backend name, or an empty Optional in the case of batch jobs

enableCheckpointing

public void enableCheckpointing(CheckpointCoordinatorConfiguration chkConfig,
                                List<ExecutionJobVertex> verticesToTrigger,
                                List<ExecutionJobVertex> verticesToWaitFor,
                                List<ExecutionJobVertex> verticesToCommitTo,
                                List<MasterTriggerRestoreHook<?>> masterHooks,
                                CheckpointIDCounter checkpointIDCounter,
                                CompletedCheckpointStore checkpointStore,
                                StateBackend checkpointStateBackend,
                                CheckpointStatsTracker statsTracker)

getCheckpointCoordinator

@Nullable
public CheckpointCoordinator getCheckpointCoordinator()

getKvStateLocationRegistry

public KvStateLocationRegistry getKvStateLocationRegistry()

getRestartStrategy

public RestartStrategy getRestartStrategy()

getCheckpointCoordinatorConfiguration

public CheckpointCoordinatorConfiguration getCheckpointCoordinatorConfiguration()

Description copied from interface: AccessExecutionGraph

Returns the CheckpointCoordinatorConfiguration or null if checkpointing is disabled.

Specified by:

getCheckpointCoordinatorConfiguration in interface AccessExecutionGraph

Returns:

JobCheckpointingConfiguration for this execution graph

getCheckpointStatsSnapshot

public CheckpointStatsSnapshot getCheckpointStatsSnapshot()

Description copied from interface: AccessExecutionGraph

Returns a snapshot of the checkpoint statistics or null if checkpointing is disabled.

Specified by:

getCheckpointStatsSnapshot in interface AccessExecutionGraph

Returns:

Snapshot of the checkpoint statistics for this execution graph

setJsonPlan

public void setJsonPlan(String jsonPlan)

getJsonPlan

public String getJsonPlan()

Description copied from interface: AccessExecutionGraph

Returns the job plan as a JSON string.

Specified by:

getJsonPlan in interface AccessExecutionGraph

Returns:

job plan as a JSON string

getSlotProviderStrategy

public SlotProviderStrategy getSlotProviderStrategy()

getJobInformationOrBlobKey

public Either<SerializedValue<JobInformation>,PermanentBlobKey> getJobInformationOrBlobKey()

getJobID

public JobID getJobID()

Description copied from interface: AccessExecutionGraph

Returns the JobID for this execution graph.

Specified by:

getJobID in interface AccessExecutionGraph

Returns:

job ID for this execution graph

getJobName

public String getJobName()

Description copied from interface: AccessExecutionGraph

Returns the job name for the execution graph.

Specified by:

getJobName in interface AccessExecutionGraph

Returns:

job name for this execution graph

isStoppable

public boolean isStoppable()

Description copied from interface: AccessExecutionGraph

Returns whether the job for this execution graph is stoppable.

Specified by:

isStoppable in interface AccessExecutionGraph

Returns:

true, if all sources tasks are stoppable, false otherwise

getJobConfiguration

public Configuration getJobConfiguration()

getUserClassLoader

public ClassLoader getUserClassLoader()

getState

public JobStatus getState()

Description copied from interface: AccessExecutionGraph

Returns the current JobStatus for this execution graph.

Specified by:

getState in interface AccessExecutionGraph

Returns:

job status for this execution graph

getFailureCause

public Throwable getFailureCause()

getFailureInfo

public ErrorInfo getFailureInfo()

Description copied from interface: AccessExecutionGraph

Returns the exception that caused the job to fail. This is the first root exception that was not recoverable and triggered job failure.

Specified by:

getFailureInfo in interface AccessExecutionGraph

Returns:

failure causing exception, or null

getNumberOfRestarts

public long getNumberOfRestarts()

Gets the number of restarts, including full restarts and fine grained restarts. If a recovery is currently pending, this recovery is included in the count.

Returns:

The number of restarts so far

getJobVertex

public ExecutionJobVertex getJobVertex(JobVertexID id)

Description copied from interface: AccessExecutionGraph

Returns the job vertex for the given JobVertexID.

Specified by:

getJobVertex in interface AccessExecutionGraph

Parameters:

id - id of job vertex to be returned

Returns:

job vertex for the given id, or null

getAllVertices

public Map<JobVertexID,ExecutionJobVertex> getAllVertices()

Description copied from interface: AccessExecutionGraph

Returns a map containing all job vertices for this execution graph.

Specified by:

getAllVertices in interface AccessExecutionGraph

Returns:

map containing all job vertices for this execution graph

getVerticesTopologically

public Iterable<ExecutionJobVertex> getVerticesTopologically()

Description copied from interface: AccessExecutionGraph

Returns an iterable containing all job vertices for this execution graph in the order they were created.

Specified by:

getVerticesTopologically in interface AccessExecutionGraph

Returns:

iterable containing all job vertices for this execution graph in the order they were created

getTotalNumberOfVertices

public int getTotalNumberOfVertices()

getAllIntermediateResults

public Map<IntermediateDataSetID,IntermediateResult> getAllIntermediateResults()

getAllExecutionVertices

public Iterable<ExecutionVertex> getAllExecutionVertices()

Description copied from interface: AccessExecutionGraph

Returns an iterable containing all execution vertices for this execution graph.

Specified by:

getAllExecutionVertices in interface AccessExecutionGraph

Returns:

iterable containing all execution vertices for this execution graph

getStatusTimestamp

public long getStatusTimestamp(JobStatus status)

Description copied from interface: AccessExecutionGraph

Returns the timestamp for the given JobStatus.

Specified by:

getStatusTimestamp in interface AccessExecutionGraph

Parameters:

status - status for which the timestamp should be returned

Returns:

timestamp for the given job status

getBlobWriter

public final BlobWriter getBlobWriter()

getFutureExecutor

public Executor getFutureExecutor()

Returns the ExecutionContext associated with this ExecutionGraph.

Returns:

ExecutionContext associated with this ExecutionGraph

aggregateUserAccumulators

public Map<String,OptionalFailure<Accumulator<?,?>>> aggregateUserAccumulators()

Merges all accumulator results from the tasks previously executed in the Executions.

Returns:

The accumulator map

getAccumulatorsSerialized

public Map<String,SerializedValue<OptionalFailure<Object>>> getAccumulatorsSerialized()

Gets a serialized accumulator map.

Specified by:

getAccumulatorsSerialized in interface AccessExecutionGraph

Returns:

The accumulator map with serialized accumulator values.

getAccumulatorResultsStringified

public StringifiedAccumulatorResult[] getAccumulatorResultsStringified()

Returns the a stringified version of the user-defined accumulators.

Specified by:

getAccumulatorResultsStringified in interface AccessExecutionGraph

Returns:

an Array containing the StringifiedAccumulatorResult objects

enableNgScheduling

public void enableNgScheduling(InternalFailuresListener internalTaskFailuresListener)

attachJobGraph

public void attachJobGraph(List<JobVertex> topologiallySorted)
                    throws JobException

Throws:

JobException

isLegacyScheduling

public boolean isLegacyScheduling()

transitionToRunning

public void transitionToRunning()

scheduleForExecution

public void scheduleForExecution()
                          throws JobException

Throws:

JobException

cancel

public void cancel()

suspend

public void suspend(Throwable suspensionCause)

Suspends the current ExecutionGraph.

The JobStatus will be directly set to JobStatus.SUSPENDED iff the current state is not a terminal state. All ExecutionJobVertices will be canceled and the onTerminalState() is executed.

The JobStatus.SUSPENDED state is a local terminal state which stops the execution of the job but does not remove the job from the HA job store so that it can be recovered by another JobManager.

Parameters:

suspensionCause - Cause of the suspension

failGlobal

public void failGlobal(Throwable t)

Fails the execution graph globally. This failure will not be recovered by a specific failover strategy, but results in a full restart of all tasks.

This global failure is meant to be triggered in cases where the consistency of the execution graph' state cannot be guaranteed any more (for example when catching unexpected exceptions that indicate a bug or an unexpected call race), and where a full restart is the safe way to get consistency back.

Parameters:

t - The exception that caused the failure.

restart

public void restart(long expectedGlobalVersion)

getArchivedExecutionConfig

public ArchivedExecutionConfig getArchivedExecutionConfig()

Returns the serializable ArchivedExecutionConfig.

Specified by:

getArchivedExecutionConfig in interface AccessExecutionGraph

Returns:

ArchivedExecutionConfig which may be null in case of errors

getTerminationFuture

public CompletableFuture<JobStatus> getTerminationFuture()

Returns the termination future of this ExecutionGraph. The termination future is completed with the terminal JobStatus once the ExecutionGraph reaches this terminal state and all Execution have been terminated.

Returns:

Termination future of this ExecutionGraph.

waitUntilTerminal

@VisibleForTesting
public JobStatus waitUntilTerminal()
                                               throws InterruptedException

Throws:

InterruptedException

getFailoverStrategy

public FailoverStrategy getFailoverStrategy()

Gets the failover strategy used by the execution graph to recover from failures of tasks.

getGlobalModVersion

public long getGlobalModVersion()

Gets the current global modification version of the ExecutionGraph. The global modification version is incremented with each global action (cancel/fail/restart) and is used to disambiguate concurrent modifications between local and global failover actions.

transitionState

public boolean transitionState(JobStatus current,
                               JobStatus newState)

incrementRestarts

public void incrementRestarts()

initFailureCause

public void initFailureCause(Throwable t)

failJob

public void failJob(Throwable cause)

updateState

public boolean updateState(TaskExecutionState state)

Updates the state of one of the ExecutionVertex's Execution attempts. If the new status if "FINISHED", this also updates the accumulators.

Parameters:

state - The state update.

Returns:

True, if the task update was properly applied, false, if the execution attempt was not found.

scheduleOrUpdateConsumers

public void scheduleOrUpdateConsumers(ResultPartitionID partitionId)
                               throws ExecutionGraphException

Schedule or updates consumers of the given result partition.

Parameters:

partitionId - specifying the result partition whose consumer shall be scheduled or updated

Throws:

ExecutionGraphException - if the schedule or update consumers operation could not be executed

getRegisteredExecutions

public Map<ExecutionAttemptID,Execution> getRegisteredExecutions()

updateAccumulators

public void updateAccumulators(AccumulatorSnapshot accumulatorSnapshot)

Updates the accumulators during the runtime of a job. Final accumulator results are transferred through the UpdateTaskExecutionState message.

Parameters:

accumulatorSnapshot - The serialized flink and user-defined accumulators

registerJobStatusListener

public void registerJobStatusListener(JobStatusListener listener)

getPartitionTracker

public JobMasterPartitionTracker getPartitionTracker()

getResultPartitionAvailabilityChecker

public ResultPartitionAvailabilityChecker getResultPartitionAvailabilityChecker()