public class JobGraph extends Object implements Serializable
The JobGraph is a graph of vertices and intermediate results that are connected together to form a DAG. Note that iterations (feedback edges) are currently not encoded inside the JobGraph but inside certain special vertices that establish the feedback channel amongst themselves.
The JobGraph defines the job-wide configuration settings, while each vertex and intermediate result define the characteristics of the concrete operation and intermediate data.
Constructor and Description |
---|
JobGraph(JobID jobId,
String jobName)
Constructs a new job graph with the given job ID (or a random ID, if
null is passed),
the given name and the given execution configuration (see ExecutionConfig ). |
JobGraph(JobID jobId,
String jobName,
JobVertex... vertices)
Constructs a new job graph with the given name, the given
ExecutionConfig ,
the given jobId or a random one if null supplied, and the given job vertices. |
JobGraph(JobVertex... vertices)
Constructs a new job graph with no name, a random job ID, the given
ExecutionConfig , and
the given job vertices. |
JobGraph(String jobName)
Constructs a new job graph with the given name, the given
ExecutionConfig ,
and a random job ID. |
JobGraph(String jobName,
JobVertex... vertices)
Constructs a new job graph with the given name, the given
ExecutionConfig , a random job ID,
and the given job vertices. |
Modifier and Type | Method and Description |
---|---|
void |
addBlob(BlobKey key)
Adds the BLOB referenced by the key to the JobGraph's dependencies.
|
void |
addJar(Path jar)
Adds the path of a JAR file required to run the job on a task manager.
|
void |
addVertex(JobVertex vertex)
Adds a new task vertex to the job graph if it is not already included.
|
JobVertex |
findVertexByID(JobVertexID id)
Searches for a vertex with a matching ID and returns it.
|
boolean |
getAllowQueuedScheduling() |
List<URL> |
getClasspaths() |
Configuration |
getJobConfiguration()
Returns the configuration object for this job.
|
JobID |
getJobID()
Returns the ID of the job.
|
int |
getMaximumParallelism()
Gets the maximum parallelism of all operations in this job graph.
|
String |
getName()
Returns the name assigned to the job graph.
|
int |
getNumberOfVertices()
Returns the number of all vertices.
|
SavepointRestoreSettings |
getSavepointRestoreSettings()
Returns the configured savepoint restore setting.
|
ScheduleMode |
getScheduleMode() |
SerializedValue<ExecutionConfig> |
getSerializedExecutionConfig()
Returns the
ExecutionConfig |
long |
getSessionTimeout()
Gets the timeout after which the corresponding ExecutionGraph is removed at the
job manager after it has been executed.
|
JobSnapshottingSettings |
getSnapshotSettings()
Gets the settings for asynchronous snapshots.
|
List<BlobKey> |
getUserJarBlobKeys()
Returns a set of BLOB keys referring to the JAR files required to run this job.
|
Iterable<JobVertex> |
getVertices()
Returns an Iterable to iterate all vertices registered with the job graph.
|
JobVertex[] |
getVerticesAsArray()
Returns an array of all job vertices that are registered with the job graph.
|
List<JobVertex> |
getVerticesSortedTopologicallyFromSources() |
boolean |
hasUsercodeJarFiles()
Checks whether the JobGraph has user code JAR files attached.
|
void |
setAllowQueuedScheduling(boolean allowQueuedScheduling) |
void |
setClasspaths(List<URL> paths)
Sets the classpaths required to run the job on a task manager.
|
void |
setExecutionConfig(ExecutionConfig executionConfig)
Sets a serialized copy of the passed ExecutionConfig.
|
void |
setSavepointRestoreSettings(SavepointRestoreSettings settings)
Sets the savepoint restore settings.
|
void |
setScheduleMode(ScheduleMode scheduleMode) |
void |
setSessionTimeout(long sessionTimeout)
Sets the timeout of the session in seconds.
|
void |
setSnapshotSettings(JobSnapshottingSettings settings)
Sets the settings for asynchronous snapshots.
|
String |
toString() |
void |
uploadRequiredJarFiles(InetSocketAddress serverAddress)
Uploads the previously added user jar file to the job manager through the job manager's BLOB server.
|
void |
uploadUserJars(ActorGateway jobManager,
scala.concurrent.duration.FiniteDuration askTimeout)
Uploads the previously added user JAR files to the job manager through
the job manager's BLOB server.
|
public JobGraph(String jobName)
ExecutionConfig
,
and a random job ID. The ExecutionConfig will be serialized and can't be modified afterwards.jobName
- The name of the job.public JobGraph(JobID jobId, String jobName)
null
is passed),
the given name and the given execution configuration (see ExecutionConfig
).
The ExecutionConfig will be serialized and can't be modified afterwards.jobId
- The id of the job. A random ID is generated, if null
is passed.jobName
- The name of the job.public JobGraph(JobVertex... vertices)
ExecutionConfig
, and
the given job vertices. The ExecutionConfig will be serialized and can't be modified afterwards.vertices
- The vertices to add to the graph.public JobGraph(String jobName, JobVertex... vertices)
ExecutionConfig
, a random job ID,
and the given job vertices. The ExecutionConfig will be serialized and can't be modified afterwards.jobName
- The name of the job.vertices
- The vertices to add to the graph.public JobGraph(JobID jobId, String jobName, JobVertex... vertices)
ExecutionConfig
,
the given jobId or a random one if null supplied, and the given job vertices.
The ExecutionConfig will be serialized and can't be modified afterwards.jobId
- The id of the job. A random ID is generated, if null
is passed.jobName
- The name of the job.vertices
- The vertices to add to the graph.public JobID getJobID()
public String getName()
public Configuration getJobConfiguration()
public SerializedValue<ExecutionConfig> getSerializedExecutionConfig()
ExecutionConfig
public long getSessionTimeout()
public void setSessionTimeout(long sessionTimeout)
sessionTimeout
- The timeout in secondspublic void setAllowQueuedScheduling(boolean allowQueuedScheduling)
public boolean getAllowQueuedScheduling()
public void setScheduleMode(ScheduleMode scheduleMode)
public ScheduleMode getScheduleMode()
public void setSavepointRestoreSettings(SavepointRestoreSettings settings)
settings
- The savepoint restore settings.public SavepointRestoreSettings getSavepointRestoreSettings()
public void setExecutionConfig(ExecutionConfig executionConfig)
executionConfig
- The ExecutionConfig to be serialized.public void addVertex(JobVertex vertex)
vertex
- the new task vertex to be addedpublic Iterable<JobVertex> getVertices()
public JobVertex[] getVerticesAsArray()
public int getNumberOfVertices()
public void setSnapshotSettings(JobSnapshottingSettings settings)
null
means that
snapshotting is not enabled.settings
- The snapshot settings, or null, to disable snapshotting.public JobSnapshottingSettings getSnapshotSettings()
public JobVertex findVertexByID(JobVertexID id)
id
- the ID of the vertex to search fornull
if no vertex with such ID could be foundpublic void setClasspaths(List<URL> paths)
paths
- paths of the directories/JAR files required to run the job on a task managerpublic List<JobVertex> getVerticesSortedTopologicallyFromSources() throws InvalidProgramException
InvalidProgramException
public void addJar(Path jar)
jar
- path of the JAR file required to run the job on a task managerpublic void addBlob(BlobKey key)
key
- path of the JAR file required to run the job on a task managerpublic boolean hasUsercodeJarFiles()
public List<BlobKey> getUserJarBlobKeys()
public void uploadRequiredJarFiles(InetSocketAddress serverAddress) throws IOException
serverAddress
- the network address of the BLOB serverIOException
- thrown if an I/O error occurs during the uploadpublic int getMaximumParallelism()
public void uploadUserJars(ActorGateway jobManager, scala.concurrent.duration.FiniteDuration askTimeout) throws IOException
jobManager
- JobManager actor gatewayaskTimeout
- Ask timeoutIOException
- Thrown, if the file upload to the JobManager failed.Copyright © 2014–2017 The Apache Software Foundation. All rights reserved.