org.apache.flink.yarn

Class YarnClusterClient

java.lang.Object

org.apache.flink.client.program.ClusterClient<org.apache.hadoop.yarn.api.records.ApplicationId>

org.apache.flink.yarn.YarnClusterClient

public class YarnClusterClient
extends ClusterClient<org.apache.hadoop.yarn.api.records.ApplicationId>

Java representation of a running Flink cluster within YARN.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.client.program.ClusterClient

ClusterClient.LazyActorSystemLoader

Field Summary

Fields inherited from class org.apache.flink.client.program.ClusterClient

actorSystemLoader, flinkConfig, highAvailabilityServices, lastJobExecutionResult, log, MAX_SLOTS_UNKNOWN, timeout

Constructor Summary

Constructors

Constructor and Description
YarnClusterClient(AbstractYarnClusterDescriptor clusterDescriptor, int numberTaskManagers, int slotsPerTaskManager, org.apache.hadoop.yarn.api.records.ApplicationReport appReport, Configuration flinkConfig, boolean newlyCreatedCluster) Create a new Flink on YARN cluster.

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.yarn.api.records.ApplicationId	getApplicationId()
org.apache.hadoop.yarn.api.records.ApplicationId	getClusterId() Returns the cluster id identifying the cluster to which the client is connected.
GetClusterStatusResponse	getClusterStatus() This method is only available if the cluster hasn't been started in detached mode.
Configuration	getFlinkConfiguration() Return the Flink configuration object.
int	getMaxSlots() The client may define an upper limit on the number of slots to use.
List<String>	getNewMessages() May return new messages from the cluster.
String	getWebInterfaceURL() Returns an URL (as a string) to the JobManager web interface.
boolean	hasUserJarsInClassPath(List<URL> userJarFiles) Returns true if the client already has the user jar and providing it again would result in duplicate uploading of the jar.
boolean	isDetached() A flag to indicate whether this clients submits jobs detached.
void	shutDownCluster()
JobSubmissionResult	submitJob(JobGraph jobGraph, ClassLoader classLoader) Calls the subclasses' submitJob method.
void	waitForClusterToBeReady() Blocks until all TaskManagers are connected to the JobManager.

Methods inherited from class org.apache.flink.client.program.ClusterClient

cancel, cancelWithSavepoint, connectToJob, disposeSavepoint, endSession, endSessions, getAccumulators, getAccumulators, getClusterConnectionInfo, getJobGraph, getJobGraph, getJobManagerGateway, getJobStatus, getOptimizedPlan, getOptimizedPlan, getOptimizedPlanAsJson, getPrintStatusDuringExecution, listJobs, logAndSysout, rescaleJob, retrieveJob, run, run, run, run, run, run, runDetached, setDetached, setPrintStatusDuringExecution, shutdown, stop, triggerSavepoint

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

YarnClusterClient

public YarnClusterClient(AbstractYarnClusterDescriptor clusterDescriptor,
                         int numberTaskManagers,
                         int slotsPerTaskManager,
                         org.apache.hadoop.yarn.api.records.ApplicationReport appReport,
                         Configuration flinkConfig,
                         boolean newlyCreatedCluster)
                  throws Exception

Create a new Flink on YARN cluster.

Parameters:

clusterDescriptor - The descriptor used at cluster creation

numberTaskManagers - The number of task managers, -1 if unknown

slotsPerTaskManager - Slots per task manager, -1 if unknown

appReport - the YARN application ID

flinkConfig - Flink configuration

newlyCreatedCluster - Indicator whether this cluster has just been created

Throws:

IOException

org.apache.hadoop.yarn.exceptions.YarnException

Exception

Method Detail

getFlinkConfiguration

public Configuration getFlinkConfiguration()

Description copied from class: ClusterClient

Return the Flink configuration object.

Overrides:

getFlinkConfiguration in class ClusterClient<org.apache.hadoop.yarn.api.records.ApplicationId>

Returns:

The Flink configuration object

getMaxSlots

public int getMaxSlots()

Description copied from class: ClusterClient

The client may define an upper limit on the number of slots to use.

Specified by:

getMaxSlots in class ClusterClient<org.apache.hadoop.yarn.api.records.ApplicationId>

Returns:

-1 (ClusterClient.MAX_SLOTS_UNKNOWN) if unknown

hasUserJarsInClassPath

public boolean hasUserJarsInClassPath(List<URL> userJarFiles)

Description copied from class: ClusterClient

Returns true if the client already has the user jar and providing it again would result in duplicate uploading of the jar.

Specified by:

hasUserJarsInClassPath in class ClusterClient<org.apache.hadoop.yarn.api.records.ApplicationId>

submitJob

public JobSubmissionResult submitJob(JobGraph jobGraph,
                                     ClassLoader classLoader)
                              throws ProgramInvocationException

Description copied from class: ClusterClient

Calls the subclasses' submitJob method. It may decide to simply call one of the run methods or it may perform some custom job submission logic.

Specified by:

submitJob in class ClusterClient<org.apache.hadoop.yarn.api.records.ApplicationId>

Parameters:

jobGraph - The JobGraph to be submitted

Returns:

JobSubmissionResult

Throws:

ProgramInvocationException

getWebInterfaceURL

public String getWebInterfaceURL()

Description copied from class: ClusterClient

Returns an URL (as a string) to the JobManager web interface.

Specified by:

getWebInterfaceURL in class ClusterClient<org.apache.hadoop.yarn.api.records.ApplicationId>

getClusterStatus

public GetClusterStatusResponse getClusterStatus()

This method is only available if the cluster hasn't been started in detached mode.

Specified by:

getClusterStatus in class ClusterClient<org.apache.hadoop.yarn.api.records.ApplicationId>

getNewMessages

public List<String> getNewMessages()

Description copied from class: ClusterClient

May return new messages from the cluster. Messages can be for example about failed containers or container launch requests.

Specified by:

getNewMessages in class ClusterClient<org.apache.hadoop.yarn.api.records.ApplicationId>

getClusterId

public org.apache.hadoop.yarn.api.records.ApplicationId getClusterId()

Description copied from class: ClusterClient

Returns the cluster id identifying the cluster to which the client is connected.

Specified by:

getClusterId in class ClusterClient<org.apache.hadoop.yarn.api.records.ApplicationId>

Returns:

cluster id of the connected cluster

isDetached

public boolean isDetached()

Description copied from class: ClusterClient

A flag to indicate whether this clients submits jobs detached.

Overrides:

isDetached in class ClusterClient<org.apache.hadoop.yarn.api.records.ApplicationId>

Returns:

True if the Client submits detached, false otherwise

waitForClusterToBeReady

public void waitForClusterToBeReady()

Blocks until all TaskManagers are connected to the JobManager.

Specified by:

waitForClusterToBeReady in class ClusterClient<org.apache.hadoop.yarn.api.records.ApplicationId>

shutDownCluster

public void shutDownCluster()

Overrides:

shutDownCluster in class ClusterClient<org.apache.hadoop.yarn.api.records.ApplicationId>

getApplicationId

public org.apache.hadoop.yarn.api.records.ApplicationId getApplicationId()