org.apache.flink.yarn

Class YarnClusterClientV2

java.lang.Object

org.apache.flink.client.program.ClusterClient

org.apache.flink.yarn.YarnClusterClientV2

public class YarnClusterClientV2
extends ClusterClient

Java representation of a running Flink job on YARN. Since flip-6, a flink job will be run as a yarn job by default, each job has a jobmaster, so this class will be used as a client to communicate with yarn and start the job on yarn.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.client.program.ClusterClient

ClusterClient.LazyActorSystemLoader

Field Summary

Fields inherited from class org.apache.flink.client.program.ClusterClient

actorSystemLoader, flinkConfig, highAvailabilityServices, log, timeout

Constructor Summary

Constructors

Constructor and Description
YarnClusterClientV2(AbstractYarnClusterDescriptor clusterDescriptor, Configuration flinkConfig) Create a client to communicate with YARN cluster.

Method Summary

Modifier and Type	Method and Description
void	finalizeCluster() Request the cluster to shut down or disconnect.
org.apache.hadoop.yarn.api.records.ApplicationId	getApplicationId()
ApplicationStatus	getApplicationStatus()
String	getClusterIdentifier() Returns a string representation of the cluster.
GetClusterStatusResponse	getClusterStatus() This method is only available if the cluster hasn't been started in detached mode.
Configuration	getFlinkConfiguration() Return the Flink configuration object.
InetSocketAddress	getJobManagerAddress() Gets the current JobManager address (may change in case of a HA setup).
int	getMaxSlots() The client may define an upper limit on the number of slots to use.
List<String>	getNewMessages() May return new messages from the cluster.
String	getWebInterfaceURL() Returns an URL (as a string) to the JobManager web interface.
boolean	hasUserJarsInClassPath(List<URL> userJarFiles) Returns true if the client already has the user jar and providing it again would result in duplicate uploading of the jar.
boolean	isDetached() A flag to indicate whether this clients submits jobs detached.
protected JobSubmissionResult	submitJob(JobGraph jobGraph, ClassLoader classLoader) Calls the subclasses' submitJob method.
void	waitForClusterToBeReady() Blocks until the client has determined that the cluster is ready for Job submission.

Methods inherited from class org.apache.flink.client.program.ClusterClient

cancel, cancelWithSavepoint, connectToJob, endSession, endSessions, getAccumulators, getAccumulators, getJobGraph, getJobManagerGateway, getOptimizedPlan, getOptimizedPlan, getOptimizedPlanAsJson, getPrintStatusDuringExecution, listJobs, logAndSysout, retrieveJob, run, run, run, run, run, run, runDetached, setDetached, setPrintStatusDuringExecution, shutdown, stop, triggerSavepoint

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

YarnClusterClientV2

public YarnClusterClientV2(AbstractYarnClusterDescriptor clusterDescriptor,
                           Configuration flinkConfig)
                    throws Exception

Create a client to communicate with YARN cluster.

Parameters:

clusterDescriptor - The descriptor used to create yarn job

flinkConfig - Flink configuration

Throws:

Exception - if the cluster client could not be created

Method Detail

getFlinkConfiguration

public Configuration getFlinkConfiguration()

Description copied from class: ClusterClient

Return the Flink configuration object.

Overrides:

getFlinkConfiguration in class ClusterClient

Returns:

The Flink configuration object

getMaxSlots

public int getMaxSlots()

Description copied from class: ClusterClient

The client may define an upper limit on the number of slots to use.

Specified by:

getMaxSlots in class ClusterClient

Returns:

-1 if unknown

hasUserJarsInClassPath

public boolean hasUserJarsInClassPath(List<URL> userJarFiles)

Description copied from class: ClusterClient

Returns true if the client already has the user jar and providing it again would result in duplicate uploading of the jar.

Specified by:

hasUserJarsInClassPath in class ClusterClient

submitJob

protected JobSubmissionResult submitJob(JobGraph jobGraph,
                                        ClassLoader classLoader)
                                 throws ProgramInvocationException

Description copied from class: ClusterClient

Calls the subclasses' submitJob method. It may decide to simply call one of the run methods or it may perform some custom job submission logic.

Specified by:

submitJob in class ClusterClient

Parameters:

jobGraph - The JobGraph to be submitted

Returns:

JobSubmissionResult

Throws:

ProgramInvocationException

getWebInterfaceURL

public String getWebInterfaceURL()

Description copied from class: ClusterClient

Returns an URL (as a string) to the JobManager web interface.

Specified by:

getWebInterfaceURL in class ClusterClient

getClusterIdentifier

public String getClusterIdentifier()

Description copied from class: ClusterClient

Returns a string representation of the cluster.

Specified by:

getClusterIdentifier in class ClusterClient

getClusterStatus

public GetClusterStatusResponse getClusterStatus()

This method is only available if the cluster hasn't been started in detached mode.

Specified by:

getClusterStatus in class ClusterClient

getApplicationStatus

public ApplicationStatus getApplicationStatus()

getNewMessages

public List<String> getNewMessages()

Description copied from class: ClusterClient

May return new messages from the cluster. Messages can be for example about failed containers or container launch requests.

Specified by:

getNewMessages in class ClusterClient

finalizeCluster

public void finalizeCluster()

Description copied from class: ClusterClient

Request the cluster to shut down or disconnect.

Specified by:

finalizeCluster in class ClusterClient

isDetached

public boolean isDetached()

Description copied from class: ClusterClient

A flag to indicate whether this clients submits jobs detached.

Overrides:

isDetached in class ClusterClient

Returns:

True if the Client submits detached, false otherwise

waitForClusterToBeReady

public void waitForClusterToBeReady()

Description copied from class: ClusterClient

Blocks until the client has determined that the cluster is ready for Job submission.

This is delayed until right before job submission to report any other errors first (e.g. invalid job definitions/errors in the user jar)

Specified by:

waitForClusterToBeReady in class ClusterClient

getJobManagerAddress

public InetSocketAddress getJobManagerAddress()

Description copied from class: ClusterClient

Gets the current JobManager address (may change in case of a HA setup).

Overrides:

getJobManagerAddress in class ClusterClient

Returns:

The address (host and port) of the leading JobManager

getApplicationId

public org.apache.hadoop.yarn.api.records.ApplicationId getApplicationId()