org.apache.flink.client.program.rest

Class RestClusterClient<T>

java.lang.Object

org.apache.flink.client.program.ClusterClient<T>

org.apache.flink.client.program.rest.RestClusterClient<T>

All Implemented Interfaces:

NewClusterClient

public class RestClusterClient<T>
extends ClusterClient<T>
implements NewClusterClient

A ClusterClient implementation that communicates via HTTP REST requests.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.client.program.ClusterClient

ClusterClient.LazyActorSystemLoader

Field Summary

Fields inherited from class org.apache.flink.client.program.ClusterClient

actorSystemLoader, flinkConfig, highAvailabilityServices, lastJobExecutionResult, log, MAX_SLOTS_UNKNOWN, timeout

Constructor Summary

Constructors

Constructor and Description
RestClusterClient(Configuration config, T clusterId)
RestClusterClient(Configuration config, T clusterId, LeaderRetrievalService webMonitorRetrievalService)

Method Summary

Modifier and Type	Method and Description
void	cancel(JobID jobID) Cancels a job identified by the job id.
String	cancelWithSavepoint(JobID jobId, String savepointDirectory) Cancels a job identified by the job id and triggers a savepoint.
CompletableFuture<Acknowledge>	disposeSavepoint(String savepointPath)
Map<String,OptionalFailure<Object>>	getAccumulators(JobID jobID, ClassLoader loader) Requests and returns the accumulators for the given job identifier.
LeaderConnectionInfo	getClusterConnectionInfo() Gets the current cluster connection info (may change in case of a HA setup).
T	getClusterId() Returns the cluster id identifying the cluster to which the client is connected.
GetClusterStatusResponse	getClusterStatus() Returns the latest cluster status, with number of Taskmanagers and slots.
CompletableFuture<JobStatus>	getJobStatus(JobID jobId) Requests the JobStatus of the job with the given JobID.
int	getMaxSlots() The client may define an upper limit on the number of slots to use.
List<String>	getNewMessages() May return new messages from the cluster.
String	getWebInterfaceURL() Returns an URL (as a string) to the JobManager web interface.
boolean	hasUserJarsInClassPath(List<URL> userJarFiles) Returns true if the client already has the user jar and providing it again would result in duplicate uploading of the jar.
CompletableFuture<Collection<JobStatusMessage>>	listJobs() Lists the currently running and finished jobs on the cluster.
CompletableFuture<JobResult>	requestJobResult(JobID jobId) Requests the JobResult for the given JobID.
CompletableFuture<Acknowledge>	rescaleJob(JobID jobId, int newParallelism) Rescales the specified job such that it will have the new parallelism.
void	shutdown() Shuts down the client.
void	shutDownCluster()
void	stop(JobID jobID) Stops a program on Flink cluster whose job-manager is configured in this client's configuration.
CompletableFuture<JobSubmissionResult>	submitJob(JobGraph jobGraph) Submits the given JobGraph to the dispatcher.
JobSubmissionResult	submitJob(JobGraph jobGraph, ClassLoader classLoader) Calls the subclasses' submitJob method.
CompletableFuture<String>	triggerSavepoint(JobID jobId, String savepointDirectory) Triggers a savepoint for the job identified by the job id.
void	waitForClusterToBeReady() Blocks until the client has determined that the cluster is ready for Job submission.

Methods inherited from class org.apache.flink.client.program.ClusterClient

connectToJob, endSession, endSessions, getAccumulators, getFlinkConfiguration, getJobGraph, getJobGraph, getJobManagerGateway, getOptimizedPlan, getOptimizedPlan, getOptimizedPlanAsJson, getPrintStatusDuringExecution, isDetached, logAndSysout, retrieveJob, run, run, run, run, run, run, runDetached, setDetached, setPrintStatusDuringExecution

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RestClusterClient

public RestClusterClient(Configuration config,
                         T clusterId)
                  throws Exception

Throws:

Exception

RestClusterClient

public RestClusterClient(Configuration config,
                         T clusterId,
                         LeaderRetrievalService webMonitorRetrievalService)
                  throws Exception

Throws:

Exception

Method Detail

shutdown

public void shutdown()

Description copied from class: ClusterClient

Shuts down the client. This stops the internal actor system and actors.

Overrides:

shutdown in class ClusterClient<T>

submitJob

public JobSubmissionResult submitJob(JobGraph jobGraph,
                                     ClassLoader classLoader)
                              throws ProgramInvocationException

Description copied from class: ClusterClient

Calls the subclasses' submitJob method. It may decide to simply call one of the run methods or it may perform some custom job submission logic.

Specified by:

submitJob in class ClusterClient<T>

Parameters:

jobGraph - The JobGraph to be submitted

Returns:

JobSubmissionResult

Throws:

ProgramInvocationException

getJobStatus

public CompletableFuture<JobStatus> getJobStatus(JobID jobId)

Description copied from class: ClusterClient

Requests the JobStatus of the job with the given JobID.

Overrides:

getJobStatus in class ClusterClient<T>

requestJobResult

public CompletableFuture<JobResult> requestJobResult(@Nonnull
                                                     JobID jobId)

Requests the JobResult for the given JobID. The method retries multiple times to poll the JobResult before giving up.

Specified by:

requestJobResult in interface NewClusterClient

Parameters:

jobId - specifying the job for which to retrieve the JobResult

Returns:

Future which is completed with the JobResult once the job has completed or with a failure if the JobResult could not be retrieved.

submitJob

public CompletableFuture<JobSubmissionResult> submitJob(@Nonnull
                                                        JobGraph jobGraph)

Submits the given JobGraph to the dispatcher.

Specified by:

submitJob in interface NewClusterClient

Parameters:

jobGraph - to submit

Returns:

Future which is completed with the submission response

stop

public void stop(JobID jobID)
          throws Exception

Description copied from class: ClusterClient

Stops a program on Flink cluster whose job-manager is configured in this client's configuration. Stopping works only for streaming programs. Be aware, that the program might continue to run for a while after sending the stop command, because after sources stopped to emit data all operators need to finish processing.

Overrides:

stop in class ClusterClient<T>

Parameters:

jobID - the job ID of the streaming program to stop

Throws:

Exception - If the job ID is invalid (ie, is unknown or refers to a batch job) or if sending the stop signal failed. That might be due to an I/O problem, ie, the job-manager is unreachable.

cancel

public void cancel(JobID jobID)
            throws Exception

Description copied from class: ClusterClient

Cancels a job identified by the job id.

Overrides:

cancel in class ClusterClient<T>

Parameters:

jobID - the job id

Throws:

Exception - In case an error occurred.

cancelWithSavepoint

public String cancelWithSavepoint(JobID jobId,
                                  @Nullable
                                  String savepointDirectory)
                           throws Exception

Description copied from class: ClusterClient

Cancels a job identified by the job id and triggers a savepoint.

Overrides:

cancelWithSavepoint in class ClusterClient<T>

Parameters:

jobId - the job id

savepointDirectory - directory the savepoint should be written to

Returns:

path where the savepoint is located

Throws:

Exception - In case an error cocurred.

triggerSavepoint

public CompletableFuture<String> triggerSavepoint(JobID jobId,
                                                  @Nullable
                                                  String savepointDirectory)

Description copied from class: ClusterClient

Triggers a savepoint for the job identified by the job id. The savepoint will be written to the given savepoint directory, or CheckpointingOptions.SAVEPOINT_DIRECTORY if it is null.

Overrides:

triggerSavepoint in class ClusterClient<T>

Parameters:

jobId - job id

savepointDirectory - directory the savepoint should be written to

Returns:

path future where the savepoint is located

getAccumulators

public Map<String,OptionalFailure<Object>> getAccumulators(JobID jobID,
                                                           ClassLoader loader)
                                                    throws Exception

Description copied from class: ClusterClient

Requests and returns the accumulators for the given job identifier. Accumulators can be requested while a is running or after it has finished.

Overrides:

getAccumulators in class ClusterClient<T>

Parameters:

jobID - The job identifier of a job.

loader - The class loader for deserializing the accumulator results.

Returns:

A Map containing the accumulator's name and its value.

Throws:

Exception

listJobs

public CompletableFuture<Collection<JobStatusMessage>> listJobs()

Description copied from class: ClusterClient

Lists the currently running and finished jobs on the cluster.

Overrides:

listJobs in class ClusterClient<T>

Returns:

future collection of running and finished jobs

getClusterId

public T getClusterId()

Description copied from class: ClusterClient

Returns the cluster id identifying the cluster to which the client is connected.

Specified by:

getClusterId in class ClusterClient<T>

Returns:

cluster id of the connected cluster

getClusterConnectionInfo

public LeaderConnectionInfo getClusterConnectionInfo()
                                              throws LeaderRetrievalException

Description copied from class: ClusterClient

Gets the current cluster connection info (may change in case of a HA setup).

Overrides:

getClusterConnectionInfo in class ClusterClient<T>

Returns:

The the connection info to the leader component of the cluster

Throws:

LeaderRetrievalException - if the leader could not be retrieved

rescaleJob

public CompletableFuture<Acknowledge> rescaleJob(JobID jobId,
                                                 int newParallelism)

Description copied from class: ClusterClient

Rescales the specified job such that it will have the new parallelism.

Overrides:

rescaleJob in class ClusterClient<T>

Parameters:

jobId - specifying the job to modify

newParallelism - specifying the new parallelism of the rescaled job

Returns:

Future which is completed once the rescaling has been completed

disposeSavepoint

public CompletableFuture<Acknowledge> disposeSavepoint(String savepointPath)

Overrides:

disposeSavepoint in class ClusterClient<T>

shutDownCluster

public void shutDownCluster()

Overrides:

shutDownCluster in class ClusterClient<T>

hasUserJarsInClassPath

public boolean hasUserJarsInClassPath(List<URL> userJarFiles)

Description copied from class: ClusterClient

Returns true if the client already has the user jar and providing it again would result in duplicate uploading of the jar.

Specified by:

hasUserJarsInClassPath in class ClusterClient<T>

waitForClusterToBeReady

public void waitForClusterToBeReady()

Description copied from class: ClusterClient

Blocks until the client has determined that the cluster is ready for Job submission.

This is delayed until right before job submission to report any other errors first (e.g. invalid job definitions/errors in the user jar)

Specified by:

waitForClusterToBeReady in class ClusterClient<T>

getWebInterfaceURL

public String getWebInterfaceURL()

Description copied from class: ClusterClient

Returns an URL (as a string) to the JobManager web interface.

Specified by:

getWebInterfaceURL in class ClusterClient<T>

getClusterStatus

public GetClusterStatusResponse getClusterStatus()

Description copied from class: ClusterClient

Returns the latest cluster status, with number of Taskmanagers and slots.

Specified by:

getClusterStatus in class ClusterClient<T>

getNewMessages

public List<String> getNewMessages()

Description copied from class: ClusterClient

May return new messages from the cluster. Messages can be for example about failed containers or container launch requests.

Specified by:

getNewMessages in class ClusterClient<T>

getMaxSlots

public int getMaxSlots()

Description copied from class: ClusterClient

The client may define an upper limit on the number of slots to use.

Specified by:

getMaxSlots in class ClusterClient<T>

Returns:

-1 (ClusterClient.MAX_SLOTS_UNKNOWN) if unknown