public class JobGraph extends Object implements Serializable
The JobGraph is a graph of vertices and intermediate results that are connected together to form a DAG. Note that iterations (feedback edges) are currently not encoded inside the JobGraph but inside certain special vertices that establish the feedback channel amongst themselves.
The JobGraph defines the job-wide configuration settings, while each vertex and intermediate result define the characteristics of the concrete operation and intermediate data.
Constructor and Description |
---|
JobGraph(JobID jobId,
String jobName)
Constructs a new job graph with the given job ID (or a random ID, if
null is passed),
the given name and the given execution configuration (see ExecutionConfig ). |
JobGraph(JobID jobId,
String jobName,
JobVertex... vertices)
Constructs a new job graph with the given name, the given
ExecutionConfig ,
the given jobId or a random one if null supplied, and the given job vertices. |
JobGraph(JobVertex... vertices)
Constructs a new job graph with no name, a random job ID, the given
ExecutionConfig , and
the given job vertices. |
JobGraph(String jobName)
Constructs a new job graph with the given name, the given
ExecutionConfig ,
and a random job ID. |
JobGraph(String jobName,
JobVertex... vertices)
Constructs a new job graph with the given name, the given
ExecutionConfig , a random job ID,
and the given job vertices. |
Modifier and Type | Method and Description |
---|---|
void |
addBlob(PermanentBlobKey key)
Adds the BLOB referenced by the key to the JobGraph's dependencies.
|
void |
addJar(Path jar)
Adds the path of a JAR file required to run the job on a task manager.
|
void |
addVertex(JobVertex vertex)
Adds a new task vertex to the job graph if it is not already included.
|
JobVertex |
findVertexByID(JobVertexID id)
Searches for a vertex with a matching ID and returns it.
|
boolean |
getAllowQueuedScheduling() |
JobCheckpointingSettings |
getCheckpointingSettings()
Gets the settings for asynchronous snapshots.
|
List<URL> |
getClasspaths() |
Configuration |
getJobConfiguration()
Returns the configuration object for this job.
|
JobID |
getJobID()
Returns the ID of the job.
|
int |
getMaximumParallelism()
Gets the maximum parallelism of all operations in this job graph.
|
String |
getName()
Returns the name assigned to the job graph.
|
int |
getNumberOfVertices()
Returns the number of all vertices.
|
SavepointRestoreSettings |
getSavepointRestoreSettings()
Returns the configured savepoint restore setting.
|
ScheduleMode |
getScheduleMode() |
SerializedValue<ExecutionConfig> |
getSerializedExecutionConfig()
Returns the
ExecutionConfig |
long |
getSessionTimeout()
Gets the timeout after which the corresponding ExecutionGraph is removed at the
job manager after it has been executed.
|
List<PermanentBlobKey> |
getUserJarBlobKeys()
Returns a set of BLOB keys referring to the JAR files required to run this job.
|
List<Path> |
getUserJars()
Gets the list of assigned user jar paths.
|
Iterable<JobVertex> |
getVertices()
Returns an Iterable to iterate all vertices registered with the job graph.
|
JobVertex[] |
getVerticesAsArray()
Returns an array of all job vertices that are registered with the job graph.
|
List<JobVertex> |
getVerticesSortedTopologicallyFromSources() |
boolean |
hasUsercodeJarFiles()
Checks whether the JobGraph has user code JAR files attached.
|
void |
setAllowQueuedScheduling(boolean allowQueuedScheduling) |
void |
setClasspaths(List<URL> paths)
Sets the classpaths required to run the job on a task manager.
|
void |
setExecutionConfig(ExecutionConfig executionConfig)
Sets the execution config.
|
void |
setSavepointRestoreSettings(SavepointRestoreSettings settings)
Sets the savepoint restore settings.
|
void |
setScheduleMode(ScheduleMode scheduleMode) |
void |
setSessionTimeout(long sessionTimeout)
Sets the timeout of the session in seconds.
|
void |
setSnapshotSettings(JobCheckpointingSettings settings)
Sets the settings for asynchronous snapshots.
|
String |
toString() |
void |
uploadUserJars(InetSocketAddress blobServerAddress,
Configuration blobClientConfig)
Uploads the previously added user JAR files to the job manager through
the job manager's BLOB server.
|
public JobGraph(String jobName)
ExecutionConfig
,
and a random job ID. The ExecutionConfig will be serialized and can't be modified afterwards.jobName
- The name of the job.public JobGraph(JobID jobId, String jobName)
null
is passed),
the given name and the given execution configuration (see ExecutionConfig
).
The ExecutionConfig will be serialized and can't be modified afterwards.jobId
- The id of the job. A random ID is generated, if null
is passed.jobName
- The name of the job.public JobGraph(JobVertex... vertices)
ExecutionConfig
, and
the given job vertices. The ExecutionConfig will be serialized and can't be modified afterwards.vertices
- The vertices to add to the graph.public JobGraph(String jobName, JobVertex... vertices)
ExecutionConfig
, a random job ID,
and the given job vertices. The ExecutionConfig will be serialized and can't be modified afterwards.jobName
- The name of the job.vertices
- The vertices to add to the graph.public JobGraph(JobID jobId, String jobName, JobVertex... vertices)
ExecutionConfig
,
the given jobId or a random one if null supplied, and the given job vertices.
The ExecutionConfig will be serialized and can't be modified afterwards.jobId
- The id of the job. A random ID is generated, if null
is passed.jobName
- The name of the job.vertices
- The vertices to add to the graph.public JobID getJobID()
public String getName()
public Configuration getJobConfiguration()
public SerializedValue<ExecutionConfig> getSerializedExecutionConfig()
ExecutionConfig
public long getSessionTimeout()
public void setSessionTimeout(long sessionTimeout)
sessionTimeout
- The timeout in secondspublic void setAllowQueuedScheduling(boolean allowQueuedScheduling)
public boolean getAllowQueuedScheduling()
public void setScheduleMode(ScheduleMode scheduleMode)
public ScheduleMode getScheduleMode()
public void setSavepointRestoreSettings(SavepointRestoreSettings settings)
settings
- The savepoint restore settings.public SavepointRestoreSettings getSavepointRestoreSettings()
public void setExecutionConfig(ExecutionConfig executionConfig) throws IOException
executionConfig
- The ExecutionConfig to be serialized.IOException
- Thrown if the serialization of the ExecutionConfig failspublic void addVertex(JobVertex vertex)
vertex
- the new task vertex to be addedpublic Iterable<JobVertex> getVertices()
public JobVertex[] getVerticesAsArray()
public int getNumberOfVertices()
public void setSnapshotSettings(JobCheckpointingSettings settings)
null
means that
snapshotting is not enabled.settings
- The snapshot settings, or null, to disable snapshotting.public JobCheckpointingSettings getCheckpointingSettings()
public JobVertex findVertexByID(JobVertexID id)
id
- the ID of the vertex to search fornull
if no vertex with such ID could be foundpublic void setClasspaths(List<URL> paths)
paths
- paths of the directories/JAR files required to run the job on a task managerpublic int getMaximumParallelism()
public List<JobVertex> getVerticesSortedTopologicallyFromSources() throws InvalidProgramException
InvalidProgramException
public void addJar(Path jar)
jar
- path of the JAR file required to run the job on a task managerpublic List<Path> getUserJars()
public void addBlob(PermanentBlobKey key)
key
- path of the JAR file required to run the job on a task managerpublic boolean hasUsercodeJarFiles()
public List<PermanentBlobKey> getUserJarBlobKeys()
public void uploadUserJars(InetSocketAddress blobServerAddress, Configuration blobClientConfig) throws IOException
blobServerAddress
- of the blob server to upload the jars toblobClientConfig
- the blob client configurationIOException
- Thrown, if the file upload to the JobManager failed.Copyright © 2014–2018 The Apache Software Foundation. All rights reserved.