org.apache.flink.runtime.taskexecutor

Class TaskExecutor

java.lang.Object

org.apache.flink.runtime.rpc.RpcEndpoint

org.apache.flink.runtime.taskexecutor.TaskExecutor

All Implemented Interfaces:

AutoCloseable, RpcGateway, TaskExecutorGateway, TaskExecutorOperatorEventGateway, TaskExecutorThreadInfoGateway, AutoCloseableAsync

public class TaskExecutor
extends RpcEndpoint
implements TaskExecutorGateway

TaskExecutor implementation. The task executor is responsible for the execution of multiple Task.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.runtime.rpc.RpcEndpoint

RpcEndpoint.MainThreadExecutor

Field Summary

Fields

Modifier and Type	Field and Description
static String	TASK_MANAGER_NAME

Fields inherited from class org.apache.flink.runtime.rpc.RpcEndpoint

log, rpcServer

Constructor Summary

Constructors

Constructor and Description

TaskExecutor(RpcService rpcService, TaskManagerConfiguration taskManagerConfiguration, HighAvailabilityServices haServices, TaskManagerServices taskExecutorServices, ExternalResourceInfoProvider externalResourceInfoProvider, HeartbeatServices heartbeatServices, TaskManagerMetricGroup taskManagerMetricGroup, String metricQueryServiceAddress, TaskExecutorBlobService taskExecutorBlobService, FatalErrorHandler fatalErrorHandler, TaskExecutorPartitionTracker partitionTracker, DelegationTokenReceiverRepository delegationTokenReceiverRepository)

Method Summary

Modifier and Type	Method and Description
CompletableFuture<Acknowledge>	abortCheckpoint(ExecutionAttemptID executionAttemptID, long checkpointId, long latestCompletedCheckpointId, long checkpointTimestamp) Abort a checkpoint for the given task.
CompletableFuture<Boolean>	canBeReleased() Checks whether the task executor can be released.
CompletableFuture<Acknowledge>	cancelTask(ExecutionAttemptID executionAttemptID, Time timeout) Cancel the given task.
CompletableFuture<Acknowledge>	confirmCheckpoint(ExecutionAttemptID executionAttemptID, long completedCheckpointId, long completedCheckpointTimestamp, long lastSubsumedCheckpointId) Confirm a checkpoint for the given task.
void	disconnectJobManager(JobID jobId, Exception cause) Disconnects the given JobManager from the TaskManager.
void	disconnectResourceManager(Exception cause) Disconnects the ResourceManager from the TaskManager.
void	freeInactiveSlots(JobID jobId, Time timeout) Frees all currently inactive slot allocated for the given job.
CompletableFuture<Acknowledge>	freeSlot(AllocationID allocationId, Throwable cause, Time timeout) Frees the slot with the given allocation ID.
CompletableFuture<Collection<PartitionWithMetrics>>	getAndRetainPartitionWithMetrics(JobID jobId) Get and retain all partitions and their metrics located on this task executor, the metrics mainly includes the meta information of partition(partition bytes, etc).
ResourceID	getResourceID()
CompletableFuture<Void>	heartbeatFromJobManager(ResourceID resourceID, AllocatedSlotReport allocatedSlotReport) Heartbeat request from the job manager.
CompletableFuture<Void>	heartbeatFromResourceManager(ResourceID resourceID) Heartbeat request from the resource manager.
void	onStart() User overridable callback which is called from RpcEndpoint.internalCallOnStart().
CompletableFuture<Void>	onStop() Called to shut down the TaskManager.
CompletableFuture<Acknowledge>	promotePartitions(JobID jobId, Set<ResultPartitionID> partitionIds) Batch promote intermediate result partitions.
CompletableFuture<Acknowledge>	releaseClusterPartitions(Collection<IntermediateDataSetID> dataSetsToRelease, Time timeout) Releases all cluster partitions belong to any of the given data sets.
void	releasePartitions(JobID jobId, Set<ResultPartitionID> partitionIds) Batch release intermediate result partitions.
CompletableFuture<TransientBlobKey>	requestFileUploadByName(String fileName, Duration timeout) Requests the file upload of the specified name to the cluster's BlobServer.
CompletableFuture<TransientBlobKey>	requestFileUploadByNameAndType(String fileName, FileType fileType, Duration timeout) Requests the file upload of the specified name and file type to the cluster's BlobServer.
CompletableFuture<TransientBlobKey>	requestFileUploadByType(FileType fileType, Time timeout) Requests the file upload of the specified type to the cluster's BlobServer.
CompletableFuture<Collection<LogInfo>>	requestLogList(Time timeout) Requests for the historical log file names on the TaskManager.
CompletableFuture<SerializableOptional<String>>	requestMetricQueryServiceAddress(Time timeout) Returns the gateway of Metric Query Service on the TaskManager.
CompletableFuture<ProfilingInfo>	requestProfiling(int duration, ProfilingInfo.ProfilingMode mode, Duration timeout) Requests the profiling from this TaskManager.
CompletableFuture<Collection<ProfilingInfo>>	requestProfilingList(Duration timeout) Requests for the historical profiling file names on the TaskManager.
CompletableFuture<Acknowledge>	requestSlot(SlotID slotId, JobID jobId, AllocationID allocationId, ResourceProfile resourceProfile, String targetAddress, ResourceManagerId resourceManagerId, Time timeout) Requests a slot from the TaskManager.
CompletableFuture<ThreadDumpInfo>	requestThreadDump(Time timeout) Requests the thread dump from this TaskManager.
CompletableFuture<TaskThreadInfoResponse>	requestThreadInfoSamples(Collection<ExecutionAttemptID> taskExecutionAttemptIds, ThreadInfoSamplesRequest requestParams, Time timeout) Request a thread info sample from the given tasks.
CompletableFuture<Acknowledge>	sendOperatorEventToTask(ExecutionAttemptID executionAttemptID, OperatorID operatorId, SerializedValue<OperatorEvent> evt) Sends an operator event to an operator in a task executed by the Task Manager (Task Executor).
CompletableFuture<Acknowledge>	submitTask(TaskDeploymentDescriptor tdd, JobMasterId jobMasterId, Time timeout) Submit a Task to the TaskExecutor.
CompletableFuture<Acknowledge>	triggerCheckpoint(ExecutionAttemptID executionAttemptID, long checkpointId, long checkpointTimestamp, CheckpointOptions checkpointOptions) Trigger the checkpoint for the given task.
CompletableFuture<Acknowledge>	updateDelegationTokens(ResourceManagerId resourceManagerId, byte[] tokens) Sends new delegation tokens to this TaskManager.
CompletableFuture<Acknowledge>	updatePartitions(ExecutionAttemptID executionAttemptID, Iterable<PartitionInfo> partitionInfos, Time timeout) Update the task where the given partitions can be found.

Methods inherited from class org.apache.flink.runtime.rpc.RpcEndpoint

callAsync, closeAsync, getAddress, getEndpointId, getHostname, getMainThreadExecutor, getMainThreadExecutor, getRpcService, getSelfGateway, getTerminationFuture, internalCallOnStart, internalCallOnStop, isRunning, registerResource, runAsync, scheduleRunAsync, scheduleRunAsync, start, stop, unregisterResource, validateRunsInMainThread

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.rpc.RpcGateway

getAddress, getHostname

Methods inherited from interface org.apache.flink.util.AutoCloseableAsync

close

Field Detail

TASK_MANAGER_NAME

public static final String TASK_MANAGER_NAME

See Also:

Constant Field Values

Constructor Detail

TaskExecutor

public TaskExecutor(RpcService rpcService,
                    TaskManagerConfiguration taskManagerConfiguration,
                    HighAvailabilityServices haServices,
                    TaskManagerServices taskExecutorServices,
                    ExternalResourceInfoProvider externalResourceInfoProvider,
                    HeartbeatServices heartbeatServices,
                    TaskManagerMetricGroup taskManagerMetricGroup,
                    @Nullable
                    String metricQueryServiceAddress,
                    TaskExecutorBlobService taskExecutorBlobService,
                    FatalErrorHandler fatalErrorHandler,
                    TaskExecutorPartitionTracker partitionTracker,
                    DelegationTokenReceiverRepository delegationTokenReceiverRepository)

Method Detail

canBeReleased

public CompletableFuture<Boolean> canBeReleased()

Description copied from interface: TaskExecutorGateway

Checks whether the task executor can be released. It cannot be released if there're unconsumed result partitions.

Specified by:

canBeReleased in interface TaskExecutorGateway

Returns:

Future flag indicating whether the task executor can be released.

requestLogList

public CompletableFuture<Collection<LogInfo>> requestLogList(Time timeout)

Description copied from interface: TaskExecutorGateway

Requests for the historical log file names on the TaskManager.

Specified by:

requestLogList in interface TaskExecutorGateway

Returns:

A Tuple2 Array with all log file names with its length.

onStart

public void onStart()
             throws Exception

Description copied from class: RpcEndpoint

User overridable callback which is called from RpcEndpoint.internalCallOnStart().

This method is called when the RpcEndpoint is being started. The method is guaranteed to be executed in the main thread context and can be used to start the rpc endpoint in the context of the rpc endpoint's main thread.

IMPORTANT: This method should never be called directly by the user.

Overrides:

onStart in class RpcEndpoint

Throws:

Exception - indicating that the rpc endpoint could not be started. If an exception occurs, then the rpc endpoint will automatically terminate.

onStop

public CompletableFuture<Void> onStop()

Called to shut down the TaskManager. The method closes all TaskManager services.

Overrides:

onStop in class RpcEndpoint

Returns:

Future which is completed once all post stop actions are completed. If an error occurs this future is completed exceptionally

requestThreadInfoSamples

public CompletableFuture<TaskThreadInfoResponse> requestThreadInfoSamples(Collection<ExecutionAttemptID> taskExecutionAttemptIds,
                                                                          ThreadInfoSamplesRequest requestParams,
                                                                          Time timeout)

Description copied from interface: TaskExecutorThreadInfoGateway

Request a thread info sample from the given tasks.

Specified by:

requestThreadInfoSamples in interface TaskExecutorThreadInfoGateway

Parameters:

taskExecutionAttemptIds - identifying the task to sample

requestParams - parameters of the request

timeout - of the request

Returns:

Future of stack trace sample response

submitTask

public CompletableFuture<Acknowledge> submitTask(TaskDeploymentDescriptor tdd,
                                                 JobMasterId jobMasterId,
                                                 Time timeout)

Description copied from interface: TaskExecutorGateway

Submit a Task to the TaskExecutor.

Specified by:

submitTask in interface TaskExecutorGateway

Parameters:

tdd - describing the task to submit

jobMasterId - identifying the submitting JobMaster

timeout - of the submit operation

Returns:

Future acknowledge of the successful operation

cancelTask

public CompletableFuture<Acknowledge> cancelTask(ExecutionAttemptID executionAttemptID,
                                                 Time timeout)

Description copied from interface: TaskExecutorGateway

Cancel the given task.

Specified by:

cancelTask in interface TaskExecutorGateway

Parameters:

executionAttemptID - identifying the task

timeout - for the cancel operation

Returns:

Future acknowledge if the task is successfully canceled

updatePartitions

public CompletableFuture<Acknowledge> updatePartitions(ExecutionAttemptID executionAttemptID,
                                                       Iterable<PartitionInfo> partitionInfos,
                                                       Time timeout)

Description copied from interface: TaskExecutorGateway

Update the task where the given partitions can be found.

Specified by:

updatePartitions in interface TaskExecutorGateway

Parameters:

executionAttemptID - identifying the task

partitionInfos - telling where the partition can be retrieved from

timeout - for the update partitions operation

Returns:

Future acknowledge if the partitions have been successfully updated

releasePartitions

public void releasePartitions(JobID jobId,
                              Set<ResultPartitionID> partitionIds)

Description copied from interface: TaskExecutorGateway

Batch release intermediate result partitions.

Specified by:

releasePartitions in interface TaskExecutorGateway

Parameters:

jobId - id of the job that the partitions belong to

partitionIds - partition ids to release

promotePartitions

public CompletableFuture<Acknowledge> promotePartitions(JobID jobId,
                                                        Set<ResultPartitionID> partitionIds)

Description copied from interface: TaskExecutorGateway

Batch promote intermediate result partitions.

Specified by:

promotePartitions in interface TaskExecutorGateway

Parameters:

jobId - id of the job that the partitions belong to

partitionIds - partition ids to release

Returns:

Future acknowledge that the partitions are successfully promoted.

releaseClusterPartitions

public CompletableFuture<Acknowledge> releaseClusterPartitions(Collection<IntermediateDataSetID> dataSetsToRelease,
                                                               Time timeout)

Description copied from interface: TaskExecutorGateway

Releases all cluster partitions belong to any of the given data sets.

Specified by:

releaseClusterPartitions in interface TaskExecutorGateway

Parameters:

dataSetsToRelease - data sets for which all cluster partitions should be released

timeout - for the partitions release operation

Returns:

Future acknowledge that the request was received

heartbeatFromJobManager

public CompletableFuture<Void> heartbeatFromJobManager(ResourceID resourceID,
                                                       AllocatedSlotReport allocatedSlotReport)

Description copied from interface: TaskExecutorGateway

Heartbeat request from the job manager.

Specified by:

heartbeatFromJobManager in interface TaskExecutorGateway

Parameters:

resourceID - unique id of the job manager

Returns:

future which is completed exceptionally if the operation fails

heartbeatFromResourceManager

public CompletableFuture<Void> heartbeatFromResourceManager(ResourceID resourceID)

Description copied from interface: TaskExecutorGateway

Heartbeat request from the resource manager.

Specified by:

heartbeatFromResourceManager in interface TaskExecutorGateway

Parameters:

resourceID - unique id of the resource manager

Returns:

future which is completed exceptionally if the operation fails

triggerCheckpoint

public CompletableFuture<Acknowledge> triggerCheckpoint(ExecutionAttemptID executionAttemptID,
                                                        long checkpointId,
                                                        long checkpointTimestamp,
                                                        CheckpointOptions checkpointOptions)

Description copied from interface: TaskExecutorGateway

Trigger the checkpoint for the given task. The checkpoint is identified by the checkpoint ID and the checkpoint timestamp.

Specified by:

triggerCheckpoint in interface TaskExecutorGateway

Parameters:

executionAttemptID - identifying the task

checkpointId - unique id for the checkpoint

checkpointTimestamp - is the timestamp when the checkpoint has been initiated

checkpointOptions - for performing the checkpoint

Returns:

Future acknowledge if the checkpoint has been successfully triggered

confirmCheckpoint

public CompletableFuture<Acknowledge> confirmCheckpoint(ExecutionAttemptID executionAttemptID,
                                                        long completedCheckpointId,
                                                        long completedCheckpointTimestamp,
                                                        long lastSubsumedCheckpointId)

Description copied from interface: TaskExecutorGateway

Confirm a checkpoint for the given task. The checkpoint is identified by the checkpoint ID and the checkpoint timestamp.

Specified by:

confirmCheckpoint in interface TaskExecutorGateway

Parameters:

executionAttemptID - identifying the task

completedCheckpointId - unique id for the completed checkpoint

completedCheckpointTimestamp - is the timestamp when the checkpoint has been initiated

lastSubsumedCheckpointId - unique id for the checkpoint to be subsumed

Returns:

Future acknowledge if the checkpoint has been successfully confirmed

abortCheckpoint

public CompletableFuture<Acknowledge> abortCheckpoint(ExecutionAttemptID executionAttemptID,
                                                      long checkpointId,
                                                      long latestCompletedCheckpointId,
                                                      long checkpointTimestamp)

Description copied from interface: TaskExecutorGateway

Abort a checkpoint for the given task. The checkpoint is identified by the checkpoint ID and the checkpoint timestamp.

Specified by:

abortCheckpoint in interface TaskExecutorGateway

Parameters:

executionAttemptID - identifying the task

checkpointId - unique id for the checkpoint

latestCompletedCheckpointId - the id of the latest completed checkpoint

checkpointTimestamp - is the timestamp when the checkpoint has been initiated

Returns:

Future acknowledge if the checkpoint has been successfully confirmed

requestSlot

public CompletableFuture<Acknowledge> requestSlot(SlotID slotId,
                                                  JobID jobId,
                                                  AllocationID allocationId,
                                                  ResourceProfile resourceProfile,
                                                  String targetAddress,
                                                  ResourceManagerId resourceManagerId,
                                                  Time timeout)

Description copied from interface: TaskExecutorGateway

Requests a slot from the TaskManager.

Specified by:

requestSlot in interface TaskExecutorGateway

Parameters:

slotId - slot id for the request

jobId - for which to request a slot

allocationId - id for the request

resourceProfile - of requested slot, used only for dynamic slot allocation and will be ignored otherwise

targetAddress - to which to offer the requested slots

resourceManagerId - current leader id of the ResourceManager

timeout - for the operation

Returns:

answer to the slot request

freeSlot

public CompletableFuture<Acknowledge> freeSlot(AllocationID allocationId,
                                               Throwable cause,
                                               Time timeout)

Description copied from interface: TaskExecutorGateway

Frees the slot with the given allocation ID.

Specified by:

freeSlot in interface TaskExecutorGateway

Parameters:

allocationId - identifying the slot to free

cause - of the freeing operation

timeout - for the operation

Returns:

Future acknowledge which is returned once the slot has been freed

freeInactiveSlots

public void freeInactiveSlots(JobID jobId,
                              Time timeout)

Description copied from interface: TaskExecutorGateway

Frees all currently inactive slot allocated for the given job.

Specified by:

freeInactiveSlots in interface TaskExecutorGateway

Parameters:

jobId - job for which all inactive slots should be released

timeout - for the operation

requestFileUploadByType

public CompletableFuture<TransientBlobKey> requestFileUploadByType(FileType fileType,
                                                                   Time timeout)

Description copied from interface: TaskExecutorGateway

Requests the file upload of the specified type to the cluster's BlobServer.

Specified by:

requestFileUploadByType in interface TaskExecutorGateway

Parameters:

fileType - to upload

timeout - for the asynchronous operation

Returns:

Future which is completed with the TransientBlobKey of the uploaded file.

requestFileUploadByName

public CompletableFuture<TransientBlobKey> requestFileUploadByName(String fileName,
                                                                   Duration timeout)

Description copied from interface: TaskExecutorGateway

Requests the file upload of the specified name to the cluster's BlobServer.

Specified by:

requestFileUploadByName in interface TaskExecutorGateway

Parameters:

fileName - to upload

timeout - for the asynchronous operation

Returns:

Future which is completed with the TransientBlobKey of the uploaded file.

requestFileUploadByNameAndType

public CompletableFuture<TransientBlobKey> requestFileUploadByNameAndType(String fileName,
                                                                          FileType fileType,
                                                                          Duration timeout)

Description copied from interface: TaskExecutorGateway

Requests the file upload of the specified name and file type to the cluster's BlobServer.

Specified by:

requestFileUploadByNameAndType in interface TaskExecutorGateway

Parameters:

fileName - to upload

fileType - to upload

timeout - for the asynchronous operation

Returns:

Future which is completed with the TransientBlobKey of the uploaded file.

requestMetricQueryServiceAddress

public CompletableFuture<SerializableOptional<String>> requestMetricQueryServiceAddress(Time timeout)

Description copied from interface: TaskExecutorGateway

Returns the gateway of Metric Query Service on the TaskManager.

Specified by:

requestMetricQueryServiceAddress in interface TaskExecutorGateway

Returns:

Future gateway of Metric Query Service on the TaskManager.

disconnectJobManager

public void disconnectJobManager(JobID jobId,
                                 Exception cause)

Description copied from interface: TaskExecutorGateway

Disconnects the given JobManager from the TaskManager.

Specified by:

disconnectJobManager in interface TaskExecutorGateway

Parameters:

jobId - JobID for which the JobManager was the leader

cause - for the disconnection from the JobManager

disconnectResourceManager

public void disconnectResourceManager(Exception cause)

Description copied from interface: TaskExecutorGateway

Disconnects the ResourceManager from the TaskManager.

Specified by:

disconnectResourceManager in interface TaskExecutorGateway

Parameters:

cause - for the disconnection from the ResourceManager

sendOperatorEventToTask

public CompletableFuture<Acknowledge> sendOperatorEventToTask(ExecutionAttemptID executionAttemptID,
                                                              OperatorID operatorId,
                                                              SerializedValue<OperatorEvent> evt)

Description copied from interface: TaskExecutorOperatorEventGateway

Sends an operator event to an operator in a task executed by the Task Manager (Task Executor).

The reception is acknowledged (future is completed) when the event has been dispatched to the AbstractInvokable.dispatchOperatorEvent(OperatorID, SerializedValue) method. It is not guaranteed that the event is processed successfully within the implementation. These cases are up to the task and event sender to handle (for example with an explicit response message upon success, or by triggering failure/recovery upon exception).

Specified by:

sendOperatorEventToTask in interface TaskExecutorGateway

Specified by:

sendOperatorEventToTask in interface TaskExecutorOperatorEventGateway

requestThreadDump

public CompletableFuture<ThreadDumpInfo> requestThreadDump(Time timeout)

Description copied from interface: TaskExecutorGateway

Requests the thread dump from this TaskManager.

Specified by:

requestThreadDump in interface TaskExecutorGateway

Parameters:

timeout - timeout for the asynchronous operation

Returns:

the ThreadDumpInfo for this TaskManager.

updateDelegationTokens

public CompletableFuture<Acknowledge> updateDelegationTokens(ResourceManagerId resourceManagerId,
                                                             byte[] tokens)

Description copied from interface: TaskExecutorGateway

Sends new delegation tokens to this TaskManager.

Specified by:

updateDelegationTokens in interface TaskExecutorGateway

Parameters:

resourceManagerId - current leader id of the ResourceManager

tokens - new tokens

Returns:

Future acknowledge of the successful operation

getAndRetainPartitionWithMetrics

public CompletableFuture<Collection<PartitionWithMetrics>> getAndRetainPartitionWithMetrics(JobID jobId)

Description copied from interface: TaskExecutorGateway

Get and retain all partitions and their metrics located on this task executor, the metrics mainly includes the meta information of partition(partition bytes, etc).

Specified by:

getAndRetainPartitionWithMetrics in interface TaskExecutorGateway

Parameters:

jobId - ID of the target job

Returns:

All partitions belong to the target job and their metrics

requestProfiling

public CompletableFuture<ProfilingInfo> requestProfiling(int duration,
                                                         ProfilingInfo.ProfilingMode mode,
                                                         Duration timeout)

Description copied from interface: TaskExecutorGateway

Requests the profiling from this TaskManager.

Specified by:

requestProfiling in interface TaskExecutorGateway

Parameters:

duration - profiling duration

mode - profiling mode ProfilingInfo.ProfilingMode

timeout - timeout for the asynchronous operation

Returns:

the ProfilingInfo for this TaskManager.

requestProfilingList

public CompletableFuture<Collection<ProfilingInfo>> requestProfilingList(Duration timeout)

Description copied from interface: TaskExecutorGateway

Requests for the historical profiling file names on the TaskManager.

Specified by:

requestProfilingList in interface TaskExecutorGateway

Returns:

A Collection with all profiling instances information.

getResourceID

public ResourceID getResourceID()