org.apache.flink.yarn

Class YarnClusterDescriptor

java.lang.Object

org.apache.flink.yarn.YarnClusterDescriptor

All Implemented Interfaces:

AutoCloseable, ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>

public class YarnClusterDescriptor
extends Object
implements ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>

The descriptor with deployment information for deploying a Flink cluster on Yarn.

Constructor Summary

Constructors

Constructor and Description
YarnClusterDescriptor(Configuration flinkConfiguration, org.apache.hadoop.yarn.conf.YarnConfiguration yarnConfiguration, org.apache.hadoop.yarn.client.api.YarnClient yarnClient, YarnClusterInformationRetriever yarnClusterInformationRetriever, boolean sharedYarnClient)

Method Summary

Modifier and Type	Method and Description
void	addShipFiles(List<File> shipFiles) Adds the given files to the list of files to ship.
void	close()
ClusterClientProvider<org.apache.hadoop.yarn.api.records.ApplicationId>	deployApplicationCluster(ClusterSpecification clusterSpecification, ApplicationConfiguration applicationConfiguration) Triggers deployment of an application cluster.
ClusterClientProvider<org.apache.hadoop.yarn.api.records.ApplicationId>	deployJobCluster(ClusterSpecification clusterSpecification, JobGraph jobGraph, boolean detached) Deploys a per-job cluster with the given job on the cluster.
ClusterClientProvider<org.apache.hadoop.yarn.api.records.ApplicationId>	deploySessionCluster(ClusterSpecification clusterSpecification) Triggers deployment of a cluster.
String	getClusterDescription() Returns a String containing details about the cluster (NodeManagers, available memory, ...).
Configuration	getFlinkConfiguration()
String	getNodeLabel()
org.apache.hadoop.yarn.client.api.YarnClient	getYarnClient()
protected String	getYarnJobClusterEntrypoint() The class to start the application master with.
protected String	getYarnSessionClusterEntrypoint() The class to start the application master with.
void	killCluster(org.apache.hadoop.yarn.api.records.ApplicationId applicationId) Terminates the cluster with the given cluster id.
static void	logDetachedClusterInformation(org.apache.hadoop.yarn.api.records.ApplicationId yarnApplicationId, org.slf4j.Logger logger)
ClusterClientProvider<org.apache.hadoop.yarn.api.records.ApplicationId>	retrieve(org.apache.hadoop.yarn.api.records.ApplicationId applicationId) Retrieves an existing Flink Cluster.
void	setLocalJarPath(org.apache.hadoop.fs.Path localJarPath)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

YarnClusterDescriptor

public YarnClusterDescriptor(Configuration flinkConfiguration,
                             org.apache.hadoop.yarn.conf.YarnConfiguration yarnConfiguration,
                             org.apache.hadoop.yarn.client.api.YarnClient yarnClient,
                             YarnClusterInformationRetriever yarnClusterInformationRetriever,
                             boolean sharedYarnClient)

Method Detail

getYarnClient

public org.apache.hadoop.yarn.client.api.YarnClient getYarnClient()

getYarnSessionClusterEntrypoint

protected String getYarnSessionClusterEntrypoint()

The class to start the application master with. This class runs the main method in case of session cluster.

getYarnJobClusterEntrypoint

protected String getYarnJobClusterEntrypoint()

The class to start the application master with. This class runs the main method in case of the job cluster.

getFlinkConfiguration

public Configuration getFlinkConfiguration()

setLocalJarPath

public void setLocalJarPath(org.apache.hadoop.fs.Path localJarPath)

addShipFiles

public void addShipFiles(List<File> shipFiles)

Adds the given files to the list of files to ship.

Note that any file matching "flink-dist*.jar" will be excluded from the upload by YarnApplicationFileUploader.registerMultipleLocalResources(Collection, String, LocalResourceType) since we upload the Flink uber jar ourselves and do not need to deploy it multiple times.

Parameters:

shipFiles - files to ship

getNodeLabel

public String getNodeLabel()

close

public void close()

Specified by:

close in interface AutoCloseable

Specified by:

close in interface ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>

retrieve

public ClusterClientProvider<org.apache.hadoop.yarn.api.records.ApplicationId> retrieve(org.apache.hadoop.yarn.api.records.ApplicationId applicationId)
                                                                                 throws ClusterRetrieveException

Description copied from interface: ClusterDescriptor

Retrieves an existing Flink Cluster.

Specified by:

retrieve in interface ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>

Parameters:

applicationId - The unique identifier of the running cluster

Returns:

Client for the cluster

Throws:

ClusterRetrieveException - if the cluster client could not be retrieved

deploySessionCluster

public ClusterClientProvider<org.apache.hadoop.yarn.api.records.ApplicationId> deploySessionCluster(ClusterSpecification clusterSpecification)
                                                                                             throws ClusterDeploymentException

Description copied from interface: ClusterDescriptor

Triggers deployment of a cluster.

Specified by:

deploySessionCluster in interface ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>

Parameters:

clusterSpecification - Cluster specification defining the cluster to deploy

Returns:

Client for the cluster

Throws:

ClusterDeploymentException - if the cluster could not be deployed

deployApplicationCluster

public ClusterClientProvider<org.apache.hadoop.yarn.api.records.ApplicationId> deployApplicationCluster(ClusterSpecification clusterSpecification,
                                                                                                        ApplicationConfiguration applicationConfiguration)
                                                                                                 throws ClusterDeploymentException

Description copied from interface: ClusterDescriptor

Triggers deployment of an application cluster. This corresponds to a cluster dedicated to the execution of a predefined application. The cluster will be created on application submission and torn down upon application termination. In addition, the main() of the application's user code will be executed on the cluster, rather than the client.

Specified by:

deployApplicationCluster in interface ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>

Parameters:

clusterSpecification - Cluster specification defining the cluster to deploy

applicationConfiguration - Application-specific configuration parameters

Returns:

Client for the cluster

Throws:

ClusterDeploymentException - if the cluster could not be deployed

deployJobCluster

public ClusterClientProvider<org.apache.hadoop.yarn.api.records.ApplicationId> deployJobCluster(ClusterSpecification clusterSpecification,
                                                                                                JobGraph jobGraph,
                                                                                                boolean detached)
                                                                                         throws ClusterDeploymentException

Description copied from interface: ClusterDescriptor

Deploys a per-job cluster with the given job on the cluster.

Specified by:

deployJobCluster in interface ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>

Parameters:

clusterSpecification - Initial cluster specification with which the Flink cluster is launched

jobGraph - JobGraph with which the job cluster is started

detached - true if the cluster should be stopped after the job completion without serving the result, otherwise false

Returns:

Cluster client to talk to the Flink cluster

Throws:

ClusterDeploymentException - if the cluster could not be deployed

killCluster

public void killCluster(org.apache.hadoop.yarn.api.records.ApplicationId applicationId)
                 throws FlinkException

Description copied from interface: ClusterDescriptor

Terminates the cluster with the given cluster id.

Specified by:

killCluster in interface ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>

Parameters:

applicationId - identifying the cluster to shut down

Throws:

FlinkException - if the cluster could not be terminated

getClusterDescription

public String getClusterDescription()

Description copied from interface: ClusterDescriptor

Returns a String containing details about the cluster (NodeManagers, available memory, ...).

Specified by:

getClusterDescription in interface ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>

logDetachedClusterInformation

public static void logDetachedClusterInformation(org.apache.hadoop.yarn.api.records.ApplicationId yarnApplicationId,
                                                 org.slf4j.Logger logger)