org.apache.flink.client.program.rest

Class RestClusterClient<T>

java.lang.Object

org.apache.flink.client.program.rest.RestClusterClient<T>

All Implemented Interfaces:

AutoCloseable, ClusterClient<T>

public class RestClusterClient<T>
extends Object
implements ClusterClient<T>

A ClusterClient implementation that communicates via HTTP REST requests.

Constructor Summary

Constructors

Constructor and Description
RestClusterClient(Configuration config, T clusterId)
RestClusterClient(Configuration config, T clusterId, ClientHighAvailabilityServicesFactory factory)

Method Summary

Modifier and Type	Method and Description
CompletableFuture<Acknowledge>	cancel(JobID jobID) Cancels a job identified by the job id.
CompletableFuture<String>	cancelWithSavepoint(JobID jobId, String savepointDirectory, SavepointFormatType formatType) Cancels a job identified by the job id and triggers a savepoint.
void	close()
CompletableFuture<Acknowledge>	disposeSavepoint(String savepointPath) Dispose the savepoint under the given path.
CompletableFuture<Map<String,Object>>	getAccumulators(JobID jobID, ClassLoader loader) Requests and returns the accumulators for the given job identifier.
T	getClusterId() Returns the cluster id identifying the cluster to which the client is connected.
CompletableFuture<ClusterOverviewWithVersion>	getClusterOverview() Get an overview of the Flink cluster.
Configuration	getFlinkConfiguration() Return the Flink configuration object.
CompletableFuture<JobDetailsInfo>	getJobDetails(JobID jobId) Requests the job details.
CompletableFuture<JobStatus>	getJobStatus(JobID jobId) Requests the JobStatus of the job with the given JobID.
String	getWebInterfaceURL() Returns an URL (as a string) to the cluster web interface.
CompletableFuture<Void>	invalidateClusterDataset(AbstractID clusterDatasetId) Invalidate the cached intermediate dataset with the given id.
CompletableFuture<Set<AbstractID>>	listCompletedClusterDatasetIds() Return a set of ids of the completed cluster datasets.
CompletableFuture<Collection<JobStatusMessage>>	listJobs() Lists the currently running and finished jobs on the cluster.
CompletableFuture<Void>	reportHeartbeat(JobID jobId, long expiredTimestamp) The client reports the heartbeat to the dispatcher for aliveness.
CompletableFuture<JobResult>	requestJobResult(JobID jobId) Requests the JobResult for the given JobID.
CompletableFuture<CoordinationResponse>	sendCoordinationRequest(JobID jobId, OperatorID operatorId, CoordinationRequest request) Sends out a request to a specified coordinator and return the response.
<M extends MessageHeaders<R,P,U>,U extends MessageParameters,R extends RequestBody,P extends ResponseBody> CompletableFuture<P>	sendRequest(M messageHeaders, U messageParameters, R request)
void	shutDownCluster() Shut down the cluster that this client communicate with.
CompletableFuture<String>	stopWithDetachedSavepoint(JobID jobId, boolean advanceToEndOfTime, String savepointDirectory, SavepointFormatType formatType) Stops a program on Flink cluster whose job-manager is configured in this client's configuration.
CompletableFuture<String>	stopWithSavepoint(JobID jobId, boolean advanceToEndOfTime, String savepointDirectory, SavepointFormatType formatType) Stops a program on Flink cluster whose job-manager is configured in this client's configuration.
CompletableFuture<String>	stopWithSavepoint(JobID jobId, boolean advanceToEndOfTime, String savepointDirectory, SavepointFormatType formatType, boolean isDetachedMode)
CompletableFuture<JobID>	submitJob(JobGraph jobGraph) Submit the given JobGraph to the cluster.
CompletableFuture<Long>	triggerCheckpoint(JobID jobId, CheckpointType checkpointType) Triggers a checkpoint for the job identified by the job id.
CompletableFuture<String>	triggerDetachedSavepoint(JobID jobId, String savepointDirectory, SavepointFormatType formatType) Triggers a detached savepoint for the job identified by the job id.
CompletableFuture<String>	triggerSavepoint(JobID jobId, String savepointDirectory, SavepointFormatType formatType) Triggers a savepoint for the job identified by the job id.
CompletableFuture<Acknowledge>	updateJobResourceRequirements(JobID jobId, JobResourceRequirements jobResourceRequirements) Update JobResourceRequirements of a given job.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.client.program.ClusterClient

getAccumulators

Constructor Detail

RestClusterClient

public RestClusterClient(Configuration config,
                         T clusterId)
                  throws Exception

Throws:

Exception

RestClusterClient

public RestClusterClient(Configuration config,
                         T clusterId,
                         ClientHighAvailabilityServicesFactory factory)
                  throws Exception

Throws:

Exception

Method Detail

getFlinkConfiguration

public Configuration getFlinkConfiguration()

Description copied from interface: ClusterClient

Return the Flink configuration object.

Specified by:

getFlinkConfiguration in interface ClusterClient<T>

Returns:

The Flink configuration object

close

public void close()

Specified by:

close in interface AutoCloseable

Specified by:

close in interface ClusterClient<T>

getJobDetails

public CompletableFuture<JobDetailsInfo> getJobDetails(JobID jobId)

Requests the job details.

Parameters:

jobId - The job id

Returns:

Job details

getJobStatus

public CompletableFuture<JobStatus> getJobStatus(JobID jobId)

Description copied from interface: ClusterClient

Requests the JobStatus of the job with the given JobID.

Specified by:

getJobStatus in interface ClusterClient<T>

requestJobResult

public CompletableFuture<JobResult> requestJobResult(@Nonnull
                                                     JobID jobId)

Requests the JobResult for the given JobID. The method retries multiple times to poll the JobResult before giving up.

Specified by:

requestJobResult in interface ClusterClient<T>

Parameters:

jobId - specifying the job for which to retrieve the JobResult

Returns:

Future which is completed with the JobResult once the job has completed or with a failure if the JobResult could not be retrieved.

submitJob

public CompletableFuture<JobID> submitJob(@Nonnull
                                          JobGraph jobGraph)

Description copied from interface: ClusterClient

Submit the given JobGraph to the cluster.

Specified by:

submitJob in interface ClusterClient<T>

Parameters:

jobGraph - to submit

Returns:

JobID of the submitted job

cancel

public CompletableFuture<Acknowledge> cancel(JobID jobID)

Description copied from interface: ClusterClient

Cancels a job identified by the job id.

Specified by:

cancel in interface ClusterClient<T>

Parameters:

jobID - the job id

stopWithSavepoint

public CompletableFuture<String> stopWithSavepoint(JobID jobId,
                                                   boolean advanceToEndOfTime,
                                                   @Nullable
                                                   String savepointDirectory,
                                                   SavepointFormatType formatType)

Description copied from interface: ClusterClient

Stops a program on Flink cluster whose job-manager is configured in this client's configuration. Stopping works only for streaming programs. Be aware, that the program might continue to run for a while after sending the stop command, because after sources stopped to emit data all operators need to finish processing.

Specified by:

stopWithSavepoint in interface ClusterClient<T>

Parameters:

jobId - the job ID of the streaming program to stop

advanceToEndOfTime - flag indicating if the source should inject a MAX_WATERMARK in the pipeline

savepointDirectory - directory the savepoint should be written to

formatType - a binary format of the savepoint

Returns:

a CompletableFuture containing the path where the savepoint is located

stopWithDetachedSavepoint

public CompletableFuture<String> stopWithDetachedSavepoint(JobID jobId,
                                                           boolean advanceToEndOfTime,
                                                           @Nullable
                                                           String savepointDirectory,
                                                           SavepointFormatType formatType)

Description copied from interface: ClusterClient

Stops a program on Flink cluster whose job-manager is configured in this client's configuration. Stopping works only for streaming programs. Be aware, that the program might continue to run for a while after sending the stop command, because after sources stopped to emit data all operators need to finish processing.

Specified by:

stopWithDetachedSavepoint in interface ClusterClient<T>

Parameters:

jobId - the job ID of the streaming program to stop

advanceToEndOfTime - flag indicating if the source should inject a MAX_WATERMARK in the pipeline

savepointDirectory - directory the savepoint should be written to

formatType - a binary format of the savepoint

Returns:

the savepoint trigger id

cancelWithSavepoint

public CompletableFuture<String> cancelWithSavepoint(JobID jobId,
                                                     @Nullable
                                                     String savepointDirectory,
                                                     SavepointFormatType formatType)

Description copied from interface: ClusterClient

Cancels a job identified by the job id and triggers a savepoint.

Specified by:

cancelWithSavepoint in interface ClusterClient<T>

Parameters:

jobId - the job id

savepointDirectory - directory the savepoint should be written to

formatType - a binary format of the savepoint

Returns:

future of path where the savepoint is located

triggerSavepoint

public CompletableFuture<String> triggerSavepoint(JobID jobId,
                                                  @Nullable
                                                  String savepointDirectory,
                                                  SavepointFormatType formatType)

Description copied from interface: ClusterClient

Triggers a savepoint for the job identified by the job id. The savepoint will be written to the given savepoint directory, or CheckpointingOptions.SAVEPOINT_DIRECTORY if it is null.

Specified by:

triggerSavepoint in interface ClusterClient<T>

Parameters:

jobId - job id

savepointDirectory - directory the savepoint should be written to

formatType - a binary format of the savepoint

Returns:

path future where the savepoint is located

triggerCheckpoint

public CompletableFuture<Long> triggerCheckpoint(JobID jobId,
                                                 CheckpointType checkpointType)

Description copied from interface: ClusterClient

Triggers a checkpoint for the job identified by the job id. The checkpoint will be written to the checkpoint directory for the job.

Specified by:

triggerCheckpoint in interface ClusterClient<T>

Parameters:

jobId - job id

checkpointType - the checkpoint type (configured / full / incremental)

triggerDetachedSavepoint

public CompletableFuture<String> triggerDetachedSavepoint(JobID jobId,
                                                          @Nullable
                                                          String savepointDirectory,
                                                          SavepointFormatType formatType)

Description copied from interface: ClusterClient

Triggers a detached savepoint for the job identified by the job id. The savepoint will be written to the given savepoint directory, or CheckpointingOptions.SAVEPOINT_DIRECTORY if it is null. Notice that: the detached savepoint will return with a savepoint trigger id instead of the path future, that means the client will return very quickly.

Specified by:

triggerDetachedSavepoint in interface ClusterClient<T>

Parameters:

jobId - job id

savepointDirectory - directory the savepoint should be written to

formatType - a binary format of the savepoint

Returns:

the savepoint trigger id

sendCoordinationRequest

public CompletableFuture<CoordinationResponse> sendCoordinationRequest(JobID jobId,
                                                                       OperatorID operatorId,
                                                                       CoordinationRequest request)

Description copied from interface: ClusterClient

Sends out a request to a specified coordinator and return the response.

Specified by:

sendCoordinationRequest in interface ClusterClient<T>

Parameters:

jobId - specifies the job which the coordinator belongs to

operatorId - specifies which coordinator to receive the request

request - the request to send

Returns:

the response from the coordinator

stopWithSavepoint

public CompletableFuture<String> stopWithSavepoint(JobID jobId,
                                                   boolean advanceToEndOfTime,
                                                   @Nullable
                                                   String savepointDirectory,
                                                   SavepointFormatType formatType,
                                                   boolean isDetachedMode)

getAccumulators

public CompletableFuture<Map<String,Object>> getAccumulators(JobID jobID,
                                                             ClassLoader loader)

Description copied from interface: ClusterClient

Requests and returns the accumulators for the given job identifier. Accumulators can be requested while a is running or after it has finished.

Specified by:

getAccumulators in interface ClusterClient<T>

Parameters:

jobID - The job identifier of a job.

loader - The class loader for deserializing the accumulator results.

Returns:

A Map containing the accumulator's name and its value.

listJobs

public CompletableFuture<Collection<JobStatusMessage>> listJobs()

Description copied from interface: ClusterClient

Lists the currently running and finished jobs on the cluster.

Specified by:

listJobs in interface ClusterClient<T>

Returns:

future collection of running and finished jobs

getClusterId

public T getClusterId()

Description copied from interface: ClusterClient

Returns the cluster id identifying the cluster to which the client is connected.

Specified by:

getClusterId in interface ClusterClient<T>

Returns:

cluster id of the connected cluster

disposeSavepoint

public CompletableFuture<Acknowledge> disposeSavepoint(String savepointPath)

Description copied from interface: ClusterClient

Dispose the savepoint under the given path.

Specified by:

disposeSavepoint in interface ClusterClient<T>

Parameters:

savepointPath - path to the savepoint to be disposed

Returns:

acknowledge future of the dispose action

listCompletedClusterDatasetIds

public CompletableFuture<Set<AbstractID>> listCompletedClusterDatasetIds()

Description copied from interface: ClusterClient

Return a set of ids of the completed cluster datasets.

Specified by:

listCompletedClusterDatasetIds in interface ClusterClient<T>

Returns:

A set of ids of the completely cached intermediate dataset.

invalidateClusterDataset

public CompletableFuture<Void> invalidateClusterDataset(AbstractID clusterDatasetId)

Description copied from interface: ClusterClient

Invalidate the cached intermediate dataset with the given id.

Specified by:

invalidateClusterDataset in interface ClusterClient<T>

Parameters:

clusterDatasetId - id of the cluster dataset to be invalidated.

Returns:

Future which will be completed when the cached dataset is invalidated.

reportHeartbeat

public CompletableFuture<Void> reportHeartbeat(JobID jobId,
                                               long expiredTimestamp)

Description copied from interface: ClusterClient

The client reports the heartbeat to the dispatcher for aliveness.

Specified by:

reportHeartbeat in interface ClusterClient<T>

Parameters:

jobId - The jobId for the client and the job.

Returns:

shutDownCluster

public void shutDownCluster()

Description copied from interface: ClusterClient

Shut down the cluster that this client communicate with.

Specified by:

shutDownCluster in interface ClusterClient<T>

updateJobResourceRequirements

public CompletableFuture<Acknowledge> updateJobResourceRequirements(JobID jobId,
                                                                    JobResourceRequirements jobResourceRequirements)

Update JobResourceRequirements of a given job.

Parameters:

jobId - jobId specifies the job for which to change the resource requirements

jobResourceRequirements - new resource requirements for the provided job

Returns:

Future which is completed upon successful operation.

getClusterOverview

public CompletableFuture<ClusterOverviewWithVersion> getClusterOverview()

Get an overview of the Flink cluster.

Returns:

Future with the cluster overview.

getWebInterfaceURL

public String getWebInterfaceURL()

Description copied from interface: ClusterClient

Returns an URL (as a string) to the cluster web interface.

Specified by:

getWebInterfaceURL in interface ClusterClient<T>

sendRequest

@VisibleForTesting
public <M extends MessageHeaders<R,P,U>,U extends MessageParameters,R extends RequestBody,P extends ResponseBody> CompletableFuture<P> sendRequest(M messageHeaders,
                                                                                                                                                                      U messageParameters,
                                                                                                                                                                      R request)