public class ExecutionGraph extends Object implements AccessExecutionGraph, Archiveable<ArchivedExecutionGraph>
ExecutionJobVertex
represents one vertex from the JobGraph (usually one operation like
"map" or "join") during execution. It holds the aggregated state of all parallel subtasks.
The ExecutionJobVertex is identified inside the graph by the JobVertexID
, which it takes
from the JobGraph's corresponding JobVertex.ExecutionVertex
represents one parallel subtask. For each ExecutionJobVertex, there are
as many ExecutionVertices as the parallelism. The ExecutionVertex is identified by
the ExecutionJobVertex and the number of the parallel subtaskExecution
is one attempt to execute a ExecutionVertex. There may be multiple Executions
for the ExecutionVertex, in case of a failure, or in the case where some data needs to be recomputed
because it is no longer available when requested by later operations. An Execution is always
identified by an ExecutionAttemptID
. All messages between the JobManager and the TaskManager
about deployment of tasks and updates in the task status always use the ExecutionAttemptID to
address the message receiver.Constructor and Description |
---|
ExecutionGraph(Executor futureExecutor,
Executor ioExecutor,
JobID jobId,
String jobName,
Configuration jobConfig,
SerializedValue<ExecutionConfig> serializedConfig,
Time timeout,
RestartStrategy restartStrategy,
List<BlobKey> requiredJarFiles,
List<URL> requiredClasspaths,
SlotProvider slotProvider,
ClassLoader userClassLoader,
MetricGroup metricGroup) |
Modifier and Type | Method and Description |
---|---|
Map<String,Accumulator<?,?>> |
aggregateUserAccumulators()
Merges all accumulator results from the tasks previously executed in the Executions.
|
ArchivedExecutionGraph |
archive() |
void |
attachJobGraph(List<JobVertex> topologiallySorted) |
void |
cancel() |
void |
disableSnaphotCheckpointing()
Disables checkpointing.
|
void |
enableSnapshotCheckpointing(long interval,
long checkpointTimeout,
long minPauseBetweenCheckpoints,
int maxConcurrentCheckpoints,
ExternalizedCheckpointSettings externalizeSettings,
List<ExecutionJobVertex> verticesToTrigger,
List<ExecutionJobVertex> verticesToWaitFor,
List<ExecutionJobVertex> verticesToCommitTo,
CheckpointIDCounter checkpointIDCounter,
CompletedCheckpointStore checkpointStore,
String checkpointDir,
CheckpointStatsTracker statsTracker) |
void |
fail(Throwable t) |
StringifiedAccumulatorResult[] |
getAccumulatorResultsStringified()
Returns the a stringified version of the user-defined accumulators.
|
Map<String,SerializedValue<Object>> |
getAccumulatorsSerialized()
Gets a serialized accumulator map.
|
Iterable<ExecutionVertex> |
getAllExecutionVertices()
Returns an iterable containing all execution vertices for this execution graph.
|
Map<IntermediateDataSetID,IntermediateResult> |
getAllIntermediateResults() |
Map<JobVertexID,ExecutionJobVertex> |
getAllVertices()
Returns a map containing all job vertices for this execution graph.
|
ArchivedExecutionConfig |
getArchivedExecutionConfig()
Returns the serializable ArchivedExecutionConfig
|
CheckpointCoordinator |
getCheckpointCoordinator()
Returns the
CheckpointCoordinator for this execution graph. |
CheckpointStatsSnapshot |
getCheckpointStatsSnapshot()
Returns a snapshot of the checkpoint statistics or
null if
checkpointing is disabled. |
Throwable |
getFailureCause() |
String |
getFailureCauseAsString()
Returns the exception that caused the job to fail.
|
Executor |
getFutureExecutor()
Returns the ExecutionContext associated with this ExecutionGraph.
|
Configuration |
getJobConfiguration() |
JobID |
getJobID()
Returns the
JobID for this execution graph. |
String |
getJobName()
Returns the job name for thie execution graph.
|
JobSnapshottingSettings |
getJobSnapshottingSettings()
Returns the
JobSnapshottingSettings or null if
checkpointing is disabled. |
ExecutionJobVertex |
getJobVertex(JobVertexID id)
Returns the job vertex for the given
JobVertexID . |
String |
getJsonPlan()
Returns the job plan as a JSON string.
|
KvStateLocationRegistry |
getKvStateLocationRegistry() |
int |
getNumberOfExecutionJobVertices()
Gets the number of job vertices currently held by this execution graph.
|
Map<ExecutionAttemptID,Execution> |
getRegisteredExecutions() |
Collection<URL> |
getRequiredClasspaths()
Returns a list of classpaths referring to the directories/JAR files required to run this job
|
Collection<BlobKey> |
getRequiredJarFiles()
Returns a list of BLOB keys referring to the JAR files required to run this job
|
RestartStrategy |
getRestartStrategy() |
ScheduleMode |
getScheduleMode() |
SerializedValue<JobInformation> |
getSerializedJobInformation() |
SlotProvider |
getSlotProvider() |
JobStatus |
getState()
Returns the current
JobStatus for this execution graph. |
long |
getStatusTimestamp(JobStatus status)
Returns the timestamp for the given
JobStatus |
ClassLoader |
getUserClassLoader() |
Iterable<ExecutionJobVertex> |
getVerticesTopologically()
Returns an iterable containing all job vertices for this execution graph in the order they were created.
|
boolean |
isArchived()
Returns whether this execution graph was archived.
|
boolean |
isQueuedSchedulingAllowed() |
boolean |
isStoppable()
Returns whether the job for this execution graph is stoppable.
|
void |
registerExecutionListener(ExecutionStatusListener listener) |
void |
registerJobStatusListener(JobStatusListener listener) |
void |
restart() |
void |
restoreLatestCheckpointedState(boolean errorIfNoCheckpoint,
boolean allowNonRestoredState)
Restores the latest checkpointed state.
|
void |
scheduleForExecution() |
void |
scheduleOrUpdateConsumers(ResultPartitionID partitionId)
Schedule or updates consumers of the given result partition.
|
void |
setJsonPlan(String jsonPlan) |
void |
setQueuedSchedulingAllowed(boolean allowed) |
void |
setScheduleMode(ScheduleMode scheduleMode) |
void |
stop() |
void |
suspend(Throwable suspensionCause)
Suspends the current ExecutionGraph.
|
void |
updateAccumulators(AccumulatorSnapshot accumulatorSnapshot)
Updates the accumulators during the runtime of a job.
|
boolean |
updateState(TaskExecutionState state)
Updates the state of one of the ExecutionVertex's Execution attempts.
|
void |
waitUntilFinished()
For testing: This waits until the job execution has finished.
|
public ExecutionGraph(Executor futureExecutor, Executor ioExecutor, JobID jobId, String jobName, Configuration jobConfig, SerializedValue<ExecutionConfig> serializedConfig, Time timeout, RestartStrategy restartStrategy, List<BlobKey> requiredJarFiles, List<URL> requiredClasspaths, SlotProvider slotProvider, ClassLoader userClassLoader, MetricGroup metricGroup) throws IOException
IOException
public int getNumberOfExecutionJobVertices()
public boolean isQueuedSchedulingAllowed()
public void setQueuedSchedulingAllowed(boolean allowed)
public void setScheduleMode(ScheduleMode scheduleMode)
public ScheduleMode getScheduleMode()
public boolean isArchived()
AccessExecutionGraph
isArchived
in interface AccessExecutionGraph
public void enableSnapshotCheckpointing(long interval, long checkpointTimeout, long minPauseBetweenCheckpoints, int maxConcurrentCheckpoints, ExternalizedCheckpointSettings externalizeSettings, List<ExecutionJobVertex> verticesToTrigger, List<ExecutionJobVertex> verticesToWaitFor, List<ExecutionJobVertex> verticesToCommitTo, CheckpointIDCounter checkpointIDCounter, CompletedCheckpointStore checkpointStore, String checkpointDir, CheckpointStatsTracker statsTracker)
public void disableSnaphotCheckpointing() throws Exception
The shutdown of the checkpoint coordinator might block. Make sure that calls to this method don't block the job manager actor and run asynchronously.
Exception
public CheckpointCoordinator getCheckpointCoordinator()
AccessExecutionGraph
CheckpointCoordinator
for this execution graph.getCheckpointCoordinator
in interface AccessExecutionGraph
null
if none is available.public KvStateLocationRegistry getKvStateLocationRegistry()
public RestartStrategy getRestartStrategy()
public JobSnapshottingSettings getJobSnapshottingSettings()
AccessExecutionGraph
JobSnapshottingSettings
or null
if
checkpointing is disabled.getJobSnapshottingSettings
in interface AccessExecutionGraph
public CheckpointStatsSnapshot getCheckpointStatsSnapshot()
AccessExecutionGraph
null
if
checkpointing is disabled.getCheckpointStatsSnapshot
in interface AccessExecutionGraph
public Collection<BlobKey> getRequiredJarFiles()
public Collection<URL> getRequiredClasspaths()
public void setJsonPlan(String jsonPlan)
public String getJsonPlan()
AccessExecutionGraph
getJsonPlan
in interface AccessExecutionGraph
public SlotProvider getSlotProvider()
public SerializedValue<JobInformation> getSerializedJobInformation()
public JobID getJobID()
AccessExecutionGraph
JobID
for this execution graph.getJobID
in interface AccessExecutionGraph
public String getJobName()
AccessExecutionGraph
getJobName
in interface AccessExecutionGraph
public boolean isStoppable()
AccessExecutionGraph
isStoppable
in interface AccessExecutionGraph
public Configuration getJobConfiguration()
public ClassLoader getUserClassLoader()
public JobStatus getState()
AccessExecutionGraph
JobStatus
for this execution graph.getState
in interface AccessExecutionGraph
public Throwable getFailureCause()
public String getFailureCauseAsString()
AccessExecutionGraph
getFailureCauseAsString
in interface AccessExecutionGraph
"(null)"
public ExecutionJobVertex getJobVertex(JobVertexID id)
AccessExecutionGraph
JobVertexID
.getJobVertex
in interface AccessExecutionGraph
id
- id of job vertex to be returnedpublic Map<JobVertexID,ExecutionJobVertex> getAllVertices()
AccessExecutionGraph
getAllVertices
in interface AccessExecutionGraph
public Iterable<ExecutionJobVertex> getVerticesTopologically()
AccessExecutionGraph
getVerticesTopologically
in interface AccessExecutionGraph
public Map<IntermediateDataSetID,IntermediateResult> getAllIntermediateResults()
public Iterable<ExecutionVertex> getAllExecutionVertices()
AccessExecutionGraph
getAllExecutionVertices
in interface AccessExecutionGraph
public long getStatusTimestamp(JobStatus status)
AccessExecutionGraph
JobStatus
getStatusTimestamp
in interface AccessExecutionGraph
status
- status for which the timestamp should be returnedpublic Executor getFutureExecutor()
public Map<String,Accumulator<?,?>> aggregateUserAccumulators()
public Map<String,SerializedValue<Object>> getAccumulatorsSerialized() throws IOException
getAccumulatorsSerialized
in interface AccessExecutionGraph
IOException
public StringifiedAccumulatorResult[] getAccumulatorResultsStringified()
getAccumulatorResultsStringified
in interface AccessExecutionGraph
public void attachJobGraph(List<JobVertex> topologiallySorted) throws JobException
JobException
public void scheduleForExecution() throws JobException
JobException
public void cancel()
public void stop() throws StoppingException
StoppingException
public void suspend(Throwable suspensionCause)
suspensionCause
- Cause of the suspensionpublic void fail(Throwable t)
public void restart()
public void restoreLatestCheckpointedState(boolean errorIfNoCheckpoint, boolean allowNonRestoredState) throws Exception
The recovery of checkpoints might block. Make sure that calls to this method don't block the job manager actor and run asynchronously.
errorIfNoCheckpoint
- Fail if there is no checkpoint availableallowNonRestoredState
- Allow to skip checkpoint state that cannot be mapped
to the the ExecutionGraph vertices (if the checkpoint contains state for a
job vertex that is not part of this ExecutionGraph).Exception
public ArchivedExecutionConfig getArchivedExecutionConfig()
getArchivedExecutionConfig
in interface AccessExecutionGraph
public void waitUntilFinished() throws InterruptedException
InterruptedException
public boolean updateState(TaskExecutionState state)
state
- The state update.public void scheduleOrUpdateConsumers(ResultPartitionID partitionId) throws ExecutionGraphException
partitionId
- specifying the result partition whose consumer shall be scheduled or updatedExecutionGraphException
- if the schedule or update consumers operation could not be executedpublic Map<ExecutionAttemptID,Execution> getRegisteredExecutions()
public void updateAccumulators(AccumulatorSnapshot accumulatorSnapshot)
accumulatorSnapshot
- The serialized flink and user-defined accumulatorspublic void registerJobStatusListener(JobStatusListener listener)
public void registerExecutionListener(ExecutionStatusListener listener)
public ArchivedExecutionGraph archive()
archive
in interface Archiveable<ArchivedExecutionGraph>
Copyright © 2014–2017 The Apache Software Foundation. All rights reserved.