Package org.apache.flink.yarn

Class YarnClusterDescriptor

  • All Implemented Interfaces:
    AutoCloseable, ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>
    public class YarnClusterDescriptor
extends Object
implements ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>
    The descriptor with deployment information for deploying a Flink cluster on Yarn.

    • Constructor Detail

      • YarnClusterDescriptor

        public YarnClusterDescriptor​(Configuration flinkConfiguration,
                             org.apache.hadoop.yarn.conf.YarnConfiguration yarnConfiguration,
                             org.apache.hadoop.yarn.client.api.YarnClient yarnClient,
                             YarnClusterInformationRetriever yarnClusterInformationRetriever,
                             boolean sharedYarnClient)

    • Method Detail

      • getYarnClient

        public org.apache.hadoop.yarn.client.api.YarnClient getYarnClient()

      • getYarnSessionClusterEntrypoint

        protected String getYarnSessionClusterEntrypoint()
        The class to start the application master with. This class runs the main method in case of session cluster.

      • getYarnJobClusterEntrypoint

        protected String getYarnJobClusterEntrypoint()
        The class to start the application master with. This class runs the main method in case of the job cluster.

      • getFlinkConfiguration

        public Configuration getFlinkConfiguration()

      • setLocalJarPath

        public void setLocalJarPath​(org.apache.hadoop.fs.Path localJarPath)

      • addShipFiles

        public void addShipFiles​(List<org.apache.hadoop.fs.Path> shipFiles)
        Adds the given files to the list of files to ship.

        Note that any file matching "flink-dist*.jar" will be excluded from the upload by YarnApplicationFileUploader.registerMultipleLocalResources(Collection, String, LocalResourceType) since we upload the Flink uber jar ourselves and do not need to deploy it multiple times.

        Parameters:
        shipFiles - files to ship

      • getNodeLabel

        public String getNodeLabel()

      • retrieve

        public ClusterClientProvider<org.apache.hadoop.yarn.api.records.ApplicationId> retrieve​(org.apache.hadoop.yarn.api.records.ApplicationId applicationId)
                                                                                 throws ClusterRetrieveException
        Description copied from interface: ClusterDescriptor
        Retrieves an existing Flink Cluster.
        Specified by:
        retrieve in interface ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>
        Parameters:
        applicationId - The unique identifier of the running cluster
        Returns:
        Client for the cluster
        Throws:
        ClusterRetrieveException - if the cluster client could not be retrieved

      • deployApplicationCluster

        public ClusterClientProvider<org.apache.hadoop.yarn.api.records.ApplicationId> deployApplicationCluster​(ClusterSpecification clusterSpecification,
                                                                                                        ApplicationConfiguration applicationConfiguration)
                                                                                                 throws ClusterDeploymentException
        Description copied from interface: ClusterDescriptor
        Triggers deployment of an application cluster. This corresponds to a cluster dedicated to the execution of a predefined application. The cluster will be created on application submission and torn down upon application termination. In addition, the main() of the application's user code will be executed on the cluster, rather than the client.
        Specified by:
        deployApplicationCluster in interface ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>
        Parameters:
        clusterSpecification - Cluster specification defining the cluster to deploy
        applicationConfiguration - Application-specific configuration parameters
        Returns:
        Client for the cluster
        Throws:
        ClusterDeploymentException - if the cluster could not be deployed

      • deployJobCluster

        public ClusterClientProvider<org.apache.hadoop.yarn.api.records.ApplicationId> deployJobCluster​(ClusterSpecification clusterSpecification,
                                                                                                JobGraph jobGraph,
                                                                                                boolean detached)
                                                                                         throws ClusterDeploymentException
        Description copied from interface: ClusterDescriptor
        Deploys a per-job cluster with the given job on the cluster.
        Specified by:
        deployJobCluster in interface ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>
        Parameters:
        clusterSpecification - Initial cluster specification with which the Flink cluster is launched
        jobGraph - JobGraph with which the job cluster is started
        detached - true if the cluster should be stopped after the job completion without serving the result, otherwise false
        Returns:
        Cluster client to talk to the Flink cluster
        Throws:
        ClusterDeploymentException - if the cluster could not be deployed

      • killCluster

        public void killCluster​(org.apache.hadoop.yarn.api.records.ApplicationId applicationId)
                 throws FlinkException
        Description copied from interface: ClusterDescriptor
        Terminates the cluster with the given cluster id.
        Specified by:
        killCluster in interface ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>
        Parameters:
        applicationId - identifying the cluster to shut down
        Throws:
        FlinkException - if the cluster could not be terminated

      • getClusterDescription

        public String getClusterDescription()
        Description copied from interface: ClusterDescriptor
        Returns a String containing details about the cluster (NodeManagers, available memory, ...).
        Specified by:
        getClusterDescription in interface ClusterDescriptor<org.apache.hadoop.yarn.api.records.ApplicationId>

      • logDetachedClusterInformation

        public static void logDetachedClusterInformation​(org.apache.hadoop.yarn.api.records.ApplicationId yarnApplicationId,
                                                 org.slf4j.Logger logger)