org.apache.flink.runtime.resourcemanager

Class ResourceManager<WorkerType extends ResourceIDRetrievable>

java.lang.Object

org.apache.flink.runtime.rpc.RpcEndpoint

org.apache.flink.runtime.rpc.FencedRpcEndpoint<ResourceManagerId>

org.apache.flink.runtime.resourcemanager.ResourceManager<WorkerType>

All Implemented Interfaces:

AutoCloseable, BlocklistListener, ClusterPartitionManager, ResourceManagerGateway, org.apache.flink.runtime.rpc.FencedRpcGateway<ResourceManagerId>, org.apache.flink.runtime.rpc.RpcGateway, DelegationTokenManager.Listener, org.apache.flink.util.AutoCloseableAsync

Direct Known Subclasses:

ActiveResourceManager, StandaloneResourceManager

public abstract class ResourceManager<WorkerType extends ResourceIDRetrievable>
extends org.apache.flink.runtime.rpc.FencedRpcEndpoint<ResourceManagerId>
implements DelegationTokenManager.Listener, ResourceManagerGateway

ResourceManager implementation. The resource manager is responsible for resource de-/allocation and bookkeeping.

It offers the following methods as part of its rpc interface to interact with him remotely:

• registerJobMaster(JobMasterId, ResourceID, String, JobID, Time) registers a JobMaster at the resource manager

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.runtime.rpc.RpcEndpoint

org.apache.flink.runtime.rpc.RpcEndpoint.MainThreadExecutor

Field Summary

Fields

Modifier and Type	Field and Description
protected BlocklistHandler	blocklistHandler
protected Executor	ioExecutor
static String	RESOURCE_MANAGER_NAME
protected ResourceManagerMetricGroup	resourceManagerMetricGroup

Fields inherited from class org.apache.flink.runtime.rpc.RpcEndpoint

log, rpcServer

Constructor Summary

Constructors

Constructor and Description

ResourceManager(org.apache.flink.runtime.rpc.RpcService rpcService, UUID leaderSessionId, ResourceID resourceId, HeartbeatServices heartbeatServices, DelegationTokenManager delegationTokenManager, SlotManager slotManager, ResourceManagerPartitionTrackerFactory clusterPartitionTrackerFactory, BlocklistHandler.Factory blocklistHandlerFactory, JobLeaderIdService jobLeaderIdService, ClusterInformation clusterInformation, org.apache.flink.runtime.rpc.FatalErrorHandler fatalErrorHandler, ResourceManagerMetricGroup resourceManagerMetricGroup, org.apache.flink.api.common.time.Time rpcTimeout, Executor ioExecutor)

Method Summary

Modifier and Type	Method and Description
protected void	closeJobManagerConnection(org.apache.flink.api.common.JobID jobId, org.apache.flink.runtime.resourcemanager.ResourceManager.ResourceRequirementHandling resourceRequirementHandling, Exception cause) This method should be called by the framework once it detects that a currently registered job manager has failed.
protected Optional<WorkerType>	closeTaskManagerConnection(ResourceID resourceID, Exception cause) This method should be called by the framework once it detects that a currently registered task executor has failed.
CompletableFuture<Acknowledge>	declareRequiredResources(JobMasterId jobMasterId, ResourceRequirements resourceRequirements, org.apache.flink.api.common.time.Time timeout) Declares the absolute resource requirements for a job.
CompletableFuture<Acknowledge>	deregisterApplication(ApplicationStatus finalStatus, String diagnostics) Cleanup application and shut down cluster.
void	disconnectJobManager(org.apache.flink.api.common.JobID jobId, org.apache.flink.api.common.JobStatus jobStatus, Exception cause) Disconnects a JobManager specified by the given resourceID from the ResourceManager.
void	disconnectTaskManager(ResourceID resourceId, Exception cause) Disconnects a TaskManager specified by the given resourceID from the ResourceManager.
CompletableFuture<List<ShuffleDescriptor>>	getClusterPartitionsShuffleDescriptors(IntermediateDataSetID intermediateDataSetID) Get the shuffle descriptors of the cluster partitions ordered by partition number.
Optional<InstanceID>	getInstanceIdByResourceId(ResourceID resourceID)
CompletableFuture<Integer>	getNumberOfRegisteredTaskManagers() Gets the currently registered number of TaskManagers.
protected abstract CompletableFuture<Void>	getReadyToServeFuture() Get the ready to serve future of the resource manager.
protected abstract ResourceAllocator	getResourceAllocator()
CompletableFuture<Void>	getStartedFuture() Completion of this future indicates that the resource manager is fully started and is ready to serve.
protected WorkerType	getWorkerByInstanceId(InstanceID instanceId)
protected abstract Optional<WorkerType>	getWorkerNodeIfAcceptRegistration(ResourceID resourceID) Get worker node if the worker resource is accepted.
CompletableFuture<Void>	heartbeatFromJobManager(ResourceID resourceID) Sends the heartbeat to resource manager from job manager.
CompletableFuture<Void>	heartbeatFromTaskManager(ResourceID resourceID, TaskExecutorHeartbeatPayload heartbeatPayload) Sends the heartbeat to resource manager from task manager.
protected abstract void	initialize() Initializes the framework specific components.
protected abstract void	internalDeregisterApplication(ApplicationStatus finalStatus, String optionalDiagnostics) The framework specific code to deregister the application.
protected void	jobLeaderLostLeadership(org.apache.flink.api.common.JobID jobId, JobMasterId oldJobMasterId)
CompletableFuture<Map<IntermediateDataSetID,DataSetMetaInfo>>	listDataSets() Returns all datasets for which partitions are being tracked.
CompletableFuture<Acknowledge>	notifyNewBlockedNodes(Collection<BlockedNode> newNodes) Notify new blocked node records.
void	notifySlotAvailable(InstanceID instanceID, SlotID slotId, AllocationID allocationId) Sent by the TaskExecutor to notify the ResourceManager that a slot has become available.
protected void	onFatalError(Throwable t) Notifies the ResourceManager that a fatal error has occurred and it cannot proceed.
void	onNewTokensObtained(byte[] tokens) Callback function when new delegation tokens obtained.
void	onStart()
CompletableFuture<Void>	onStop()
protected void	onWorkerRegistered(WorkerType worker, WorkerResourceSpec workerResourceSpec)
CompletableFuture<RegistrationResponse>	registerJobMaster(JobMasterId jobMasterId, ResourceID jobManagerResourceId, String jobManagerAddress, org.apache.flink.api.common.JobID jobId, org.apache.flink.api.common.time.Time timeout) Register a JobMaster at the resource manager.
protected void	registerMetrics()
CompletableFuture<RegistrationResponse>	registerTaskExecutor(TaskExecutorRegistration taskExecutorRegistration, org.apache.flink.api.common.time.Time timeout) Register a TaskExecutor at the resource manager.
CompletableFuture<Void>	releaseClusterPartitions(IntermediateDataSetID dataSetId) Releases all partitions associated with the given dataset.
protected void	removeJob(org.apache.flink.api.common.JobID jobId, Exception cause)
CompletableFuture<Void>	reportClusterPartitions(ResourceID taskExecutorId, ClusterPartitionReport clusterPartitionReport) Report the cluster partitions status in the task executor.
CompletableFuture<ProfilingInfo>	requestProfiling(ResourceID taskManagerId, int duration, ProfilingInfo.ProfilingMode mode, java.time.Duration timeout) Requests the profiling instance from the given TaskExecutor.
CompletableFuture<ResourceOverview>	requestResourceOverview(org.apache.flink.api.common.time.Time timeout) Requests the resource overview.
CompletableFuture<TaskExecutorThreadInfoGateway>	requestTaskExecutorThreadInfoGateway(ResourceID taskManagerId, org.apache.flink.api.common.time.Time timeout) Requests the TaskExecutorGateway.
CompletableFuture<TaskManagerInfoWithSlots>	requestTaskManagerDetailsInfo(ResourceID resourceId, org.apache.flink.api.common.time.Time timeout) Requests detail information about the given TaskExecutor.
CompletableFuture<TransientBlobKey>	requestTaskManagerFileUploadByName(ResourceID taskManagerId, String fileName, org.apache.flink.api.common.time.Time timeout) Request the file upload from the given TaskExecutor to the cluster's BlobServer.
CompletableFuture<TransientBlobKey>	requestTaskManagerFileUploadByNameAndType(ResourceID taskManagerId, String fileName, FileType fileType, java.time.Duration timeout) Request the file upload from the given TaskExecutor to the cluster's BlobServer.
CompletableFuture<TransientBlobKey>	requestTaskManagerFileUploadByType(ResourceID taskManagerId, FileType fileType, org.apache.flink.api.common.time.Time timeout) Request the file upload from the given TaskExecutor to the cluster's BlobServer.
CompletableFuture<Collection<TaskManagerInfo>>	requestTaskManagerInfo(org.apache.flink.api.common.time.Time timeout) Requests information about the registered TaskExecutor.
CompletableFuture<Collection<LogInfo>>	requestTaskManagerLogList(ResourceID taskManagerId, org.apache.flink.api.common.time.Time timeout) Request log list from the given TaskExecutor.
CompletableFuture<Collection<org.apache.flink.api.java.tuple.Tuple2<ResourceID,String>>>	requestTaskManagerMetricQueryServiceAddresses(org.apache.flink.api.common.time.Time timeout) Requests the paths for the TaskManager's MetricQueryService to query.
CompletableFuture<Collection<ProfilingInfo>>	requestTaskManagerProfilingList(ResourceID taskManagerId, java.time.Duration timeout) Request profiling list from the given TaskExecutor.
CompletableFuture<ThreadDumpInfo>	requestThreadDump(ResourceID taskManagerId, org.apache.flink.api.common.time.Time timeout) Requests the thread dump from the given TaskExecutor.
CompletableFuture<Acknowledge>	sendSlotReport(ResourceID taskManagerResourceId, InstanceID taskManagerRegistrationId, SlotReport slotReport, org.apache.flink.api.common.time.Time timeout) Sends the given SlotReport to the ResourceManager.
protected void	setFailUnfulfillableRequest(boolean failUnfulfillableRequest) Set SlotManager whether to fail unfulfillable slot requests.
void	stopWorkerIfSupported(WorkerType worker) Stops the given worker if supported.
protected abstract void	terminate() Terminates the framework specific components.

Methods inherited from class org.apache.flink.runtime.rpc.FencedRpcEndpoint

getFencingToken

Methods inherited from class org.apache.flink.runtime.rpc.RpcEndpoint

callAsync, closeAsync, getAddress, getEndpointId, getHostname, getMainThreadExecutor, getRpcService, getSelfGateway, getTerminationFuture, internalCallOnStart, internalCallOnStop, isRunning, registerResource, runAsync, scheduleRunAsync, scheduleRunAsync, start, stop, unregisterResource, validateRunsInMainThread

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.rpc.FencedRpcGateway

getFencingToken

Methods inherited from interface org.apache.flink.runtime.rpc.RpcGateway

getAddress, getHostname

Methods inherited from interface org.apache.flink.util.AutoCloseableAsync

close

Field Detail

RESOURCE_MANAGER_NAME

public static final String RESOURCE_MANAGER_NAME

See Also:

Constant Field Values

resourceManagerMetricGroup

protected final ResourceManagerMetricGroup resourceManagerMetricGroup

ioExecutor

protected final Executor ioExecutor

blocklistHandler

protected final BlocklistHandler blocklistHandler

Constructor Detail

ResourceManager

public ResourceManager(org.apache.flink.runtime.rpc.RpcService rpcService,
                       UUID leaderSessionId,
                       ResourceID resourceId,
                       HeartbeatServices heartbeatServices,
                       DelegationTokenManager delegationTokenManager,
                       SlotManager slotManager,
                       ResourceManagerPartitionTrackerFactory clusterPartitionTrackerFactory,
                       BlocklistHandler.Factory blocklistHandlerFactory,
                       JobLeaderIdService jobLeaderIdService,
                       ClusterInformation clusterInformation,
                       org.apache.flink.runtime.rpc.FatalErrorHandler fatalErrorHandler,
                       ResourceManagerMetricGroup resourceManagerMetricGroup,
                       org.apache.flink.api.common.time.Time rpcTimeout,
                       Executor ioExecutor)

Method Detail

onStart

public final void onStart()
                   throws Exception

Overrides:

onStart in class org.apache.flink.runtime.rpc.RpcEndpoint

Throws:

Exception

getStartedFuture

public CompletableFuture<Void> getStartedFuture()

Completion of this future indicates that the resource manager is fully started and is ready to serve.

onStop

public final CompletableFuture<Void> onStop()

Overrides:

onStop in class org.apache.flink.runtime.rpc.RpcEndpoint

registerJobMaster

public CompletableFuture<RegistrationResponse> registerJobMaster(JobMasterId jobMasterId,
                                                                 ResourceID jobManagerResourceId,
                                                                 String jobManagerAddress,
                                                                 org.apache.flink.api.common.JobID jobId,
                                                                 org.apache.flink.api.common.time.Time timeout)

Description copied from interface: ResourceManagerGateway

Register a JobMaster at the resource manager.

Specified by:

registerJobMaster in interface ResourceManagerGateway

Parameters:

jobMasterId - The fencing token for the JobMaster leader

jobManagerResourceId - The resource ID of the JobMaster that registers

jobManagerAddress - The address of the JobMaster that registers

jobId - The Job ID of the JobMaster that registers

timeout - Timeout for the future to complete

Returns:

Future registration response

registerTaskExecutor

public CompletableFuture<RegistrationResponse> registerTaskExecutor(TaskExecutorRegistration taskExecutorRegistration,
                                                                    org.apache.flink.api.common.time.Time timeout)

Description copied from interface: ResourceManagerGateway

Register a TaskExecutor at the resource manager.

Specified by:

registerTaskExecutor in interface ResourceManagerGateway

Parameters:

taskExecutorRegistration - the task executor registration.

timeout - The timeout for the response.

Returns:

The future to the response by the ResourceManager.

sendSlotReport

public CompletableFuture<Acknowledge> sendSlotReport(ResourceID taskManagerResourceId,
                                                     InstanceID taskManagerRegistrationId,
                                                     SlotReport slotReport,
                                                     org.apache.flink.api.common.time.Time timeout)

Description copied from interface: ResourceManagerGateway

Sends the given SlotReport to the ResourceManager.

Specified by:

sendSlotReport in interface ResourceManagerGateway

taskManagerRegistrationId - id identifying the sending TaskManager

slotReport - which is sent to the ResourceManager

timeout - for the operation

Returns:

Future which is completed with Acknowledge once the slot report has been received.

onWorkerRegistered

protected void onWorkerRegistered(WorkerType worker,
                                  WorkerResourceSpec workerResourceSpec)

heartbeatFromTaskManager

public CompletableFuture<Void> heartbeatFromTaskManager(ResourceID resourceID,
                                                        TaskExecutorHeartbeatPayload heartbeatPayload)

Description copied from interface: ResourceManagerGateway

Sends the heartbeat to resource manager from task manager.

Specified by:

heartbeatFromTaskManager in interface ResourceManagerGateway

Parameters:

resourceID - unique id of the task manager

heartbeatPayload - payload from the originating TaskManager

Returns:

future which is completed exceptionally if the operation fails

heartbeatFromJobManager

public CompletableFuture<Void> heartbeatFromJobManager(ResourceID resourceID)

Description copied from interface: ResourceManagerGateway

Sends the heartbeat to resource manager from job manager.

Specified by:

heartbeatFromJobManager in interface ResourceManagerGateway

Parameters:

resourceID - unique id of the job manager

Returns:

future which is completed exceptionally if the operation fails

disconnectTaskManager

public void disconnectTaskManager(ResourceID resourceId,
                                  Exception cause)

Description copied from interface: ResourceManagerGateway

Disconnects a TaskManager specified by the given resourceID from the ResourceManager.

Specified by:

disconnectTaskManager in interface ResourceManagerGateway

Parameters:

resourceId - identifying the TaskManager to disconnect

cause - for the disconnection of the TaskManager

disconnectJobManager

public void disconnectJobManager(org.apache.flink.api.common.JobID jobId,
                                 org.apache.flink.api.common.JobStatus jobStatus,
                                 Exception cause)

Description copied from interface: ResourceManagerGateway

Disconnects a JobManager specified by the given resourceID from the ResourceManager.

Specified by:

disconnectJobManager in interface ResourceManagerGateway

Parameters:

jobId - JobID for which the JobManager was the leader

jobStatus - status of the job at the time of disconnection

cause - for the disconnection of the JobManager

declareRequiredResources

public CompletableFuture<Acknowledge> declareRequiredResources(JobMasterId jobMasterId,
                                                               ResourceRequirements resourceRequirements,
                                                               org.apache.flink.api.common.time.Time timeout)

Description copied from interface: ResourceManagerGateway

Declares the absolute resource requirements for a job.

Specified by:

declareRequiredResources in interface ResourceManagerGateway

Parameters:

jobMasterId - id of the JobMaster

resourceRequirements - resource requirements

Returns:

The confirmation that the requirements have been processed

notifySlotAvailable

public void notifySlotAvailable(InstanceID instanceID,
                                SlotID slotId,
                                AllocationID allocationId)

Description copied from interface: ResourceManagerGateway

Sent by the TaskExecutor to notify the ResourceManager that a slot has become available.

Specified by:

notifySlotAvailable in interface ResourceManagerGateway

Parameters:

instanceID - TaskExecutor's instance id

slotId - The SlotID of the freed slot

allocationId - to which the slot has been allocated

deregisterApplication

public CompletableFuture<Acknowledge> deregisterApplication(ApplicationStatus finalStatus,
                                                            @Nullable
                                                            String diagnostics)

Cleanup application and shut down cluster.

Specified by:

deregisterApplication in interface ResourceManagerGateway

Parameters:

finalStatus - of the Flink application

diagnostics - diagnostics message for the Flink application or null

getNumberOfRegisteredTaskManagers

public CompletableFuture<Integer> getNumberOfRegisteredTaskManagers()

Description copied from interface: ResourceManagerGateway

Gets the currently registered number of TaskManagers.

Specified by:

getNumberOfRegisteredTaskManagers in interface ResourceManagerGateway

Returns:

The future to the number of registered TaskManagers.

requestTaskManagerInfo

public CompletableFuture<Collection<TaskManagerInfo>> requestTaskManagerInfo(org.apache.flink.api.common.time.Time timeout)

Description copied from interface: ResourceManagerGateway

Requests information about the registered TaskExecutor.

Specified by:

requestTaskManagerInfo in interface ResourceManagerGateway

Parameters:

timeout - of the request

Returns:

Future collection of TaskManager information

requestTaskManagerDetailsInfo

public CompletableFuture<TaskManagerInfoWithSlots> requestTaskManagerDetailsInfo(ResourceID resourceId,
                                                                                 org.apache.flink.api.common.time.Time timeout)

Description copied from interface: ResourceManagerGateway

Requests detail information about the given TaskExecutor.

Specified by:

requestTaskManagerDetailsInfo in interface ResourceManagerGateway

Parameters:

resourceId - identifying the TaskExecutor for which to return information

timeout - of the request

Returns:

Future TaskManager information and its allocated slots

requestResourceOverview

public CompletableFuture<ResourceOverview> requestResourceOverview(org.apache.flink.api.common.time.Time timeout)

Description copied from interface: ResourceManagerGateway

Requests the resource overview. The resource overview provides information about the connected TaskManagers, the total number of slots and the number of available slots.

Specified by:

requestResourceOverview in interface ResourceManagerGateway

Parameters:

timeout - of the request

Returns:

Future containing the resource overview

requestTaskManagerMetricQueryServiceAddresses

public CompletableFuture<Collection<org.apache.flink.api.java.tuple.Tuple2<ResourceID,String>>> requestTaskManagerMetricQueryServiceAddresses(org.apache.flink.api.common.time.Time timeout)

Description copied from interface: ResourceManagerGateway

Requests the paths for the TaskManager's MetricQueryService to query.

Specified by:

requestTaskManagerMetricQueryServiceAddresses in interface ResourceManagerGateway

Parameters:

timeout - for the asynchronous operation

Returns:

Future containing the collection of resource ids and the corresponding metric query service path

requestTaskManagerFileUploadByType

public CompletableFuture<TransientBlobKey> requestTaskManagerFileUploadByType(ResourceID taskManagerId,
                                                                              FileType fileType,
                                                                              org.apache.flink.api.common.time.Time timeout)

Description copied from interface: ResourceManagerGateway

Request the file upload from the given TaskExecutor to the cluster's BlobServer. The corresponding TransientBlobKey is returned.

Specified by:

requestTaskManagerFileUploadByType in interface ResourceManagerGateway

Parameters:

taskManagerId - identifying the TaskExecutor to upload the specified file

fileType - type of the file to upload

timeout - for the asynchronous operation

Returns:

Future which is completed with the TransientBlobKey after uploading the file to the BlobServer.

requestTaskManagerFileUploadByName

public CompletableFuture<TransientBlobKey> requestTaskManagerFileUploadByName(ResourceID taskManagerId,
                                                                              String fileName,
                                                                              org.apache.flink.api.common.time.Time timeout)

Description copied from interface: ResourceManagerGateway

Request the file upload from the given TaskExecutor to the cluster's BlobServer. The corresponding TransientBlobKey is returned. To support different type file upload with name, using requestTaskManagerFileUploadByNameAndType(org.apache.flink.runtime.clusterframework.types.ResourceID, java.lang.String, org.apache.flink.runtime.taskexecutor.FileType, java.time.Duration) as instead.

Specified by:

requestTaskManagerFileUploadByName in interface ResourceManagerGateway

Parameters:

taskManagerId - identifying the TaskExecutor to upload the specified file

fileName - name of the file to upload

timeout - for the asynchronous operation

Returns:

Future which is completed with the TransientBlobKey after uploading the file to the BlobServer.

requestTaskManagerFileUploadByNameAndType

public CompletableFuture<TransientBlobKey> requestTaskManagerFileUploadByNameAndType(ResourceID taskManagerId,
                                                                                     String fileName,
                                                                                     FileType fileType,
                                                                                     java.time.Duration timeout)

Description copied from interface: ResourceManagerGateway

Request the file upload from the given TaskExecutor to the cluster's BlobServer. The corresponding TransientBlobKey is returned.

Specified by:

requestTaskManagerFileUploadByNameAndType in interface ResourceManagerGateway

Parameters:

taskManagerId - identifying the TaskExecutor to upload the specified file

fileName - name of the file to upload

fileType - type of the file to upload

timeout - for the asynchronous operation

Returns:

Future which is completed with the TransientBlobKey after uploading the file to the BlobServer.

requestTaskManagerLogList

public CompletableFuture<Collection<LogInfo>> requestTaskManagerLogList(ResourceID taskManagerId,
                                                                        org.apache.flink.api.common.time.Time timeout)

Description copied from interface: ResourceManagerGateway

Request log list from the given TaskExecutor.

Specified by:

requestTaskManagerLogList in interface ResourceManagerGateway

Parameters:

taskManagerId - identifying the TaskExecutor to get log list from

timeout - for the asynchronous operation

Returns:

Future which is completed with the historical log list

releaseClusterPartitions

public CompletableFuture<Void> releaseClusterPartitions(IntermediateDataSetID dataSetId)

Description copied from interface: ClusterPartitionManager

Releases all partitions associated with the given dataset.

Specified by:

releaseClusterPartitions in interface ClusterPartitionManager

Parameters:

dataSetId - dataset for which all associated partitions should be released

Returns:

future that is completed once all partitions have been released

reportClusterPartitions

public CompletableFuture<Void> reportClusterPartitions(ResourceID taskExecutorId,
                                                       ClusterPartitionReport clusterPartitionReport)

Description copied from interface: ClusterPartitionManager

Report the cluster partitions status in the task executor.

Specified by:

reportClusterPartitions in interface ClusterPartitionManager

Parameters:

taskExecutorId - The id of the task executor.

clusterPartitionReport - The status of the cluster partitions.

Returns:

future that is completed once the report have been processed.

getClusterPartitionsShuffleDescriptors

public CompletableFuture<List<ShuffleDescriptor>> getClusterPartitionsShuffleDescriptors(IntermediateDataSetID intermediateDataSetID)

Description copied from interface: ClusterPartitionManager

Get the shuffle descriptors of the cluster partitions ordered by partition number.

Specified by:

getClusterPartitionsShuffleDescriptors in interface ClusterPartitionManager

Parameters:

intermediateDataSetID - The id of the dataset.

Returns:

shuffle descriptors of the cluster partitions.

listDataSets

public CompletableFuture<Map<IntermediateDataSetID,DataSetMetaInfo>> listDataSets()

Description copied from interface: ClusterPartitionManager

Returns all datasets for which partitions are being tracked.

Specified by:

listDataSets in interface ClusterPartitionManager

Returns:

tracked datasets

requestThreadDump

public CompletableFuture<ThreadDumpInfo> requestThreadDump(ResourceID taskManagerId,
                                                           org.apache.flink.api.common.time.Time timeout)

Description copied from interface: ResourceManagerGateway

Requests the thread dump from the given TaskExecutor.

Specified by:

requestThreadDump in interface ResourceManagerGateway

Parameters:

taskManagerId - taskManagerId identifying the TaskExecutor to get the thread dump from

timeout - timeout of the asynchronous operation

Returns:

Future containing the thread dump information

requestTaskManagerProfilingList

public CompletableFuture<Collection<ProfilingInfo>> requestTaskManagerProfilingList(ResourceID taskManagerId,
                                                                                    java.time.Duration timeout)

Description copied from interface: ResourceManagerGateway

Request profiling list from the given TaskExecutor.

Specified by:

requestTaskManagerProfilingList in interface ResourceManagerGateway

Parameters:

taskManagerId - identifying the TaskExecutor to get profiling list from

timeout - for the asynchronous operation

Returns:

Future which is completed with the historical profiling list

requestProfiling

public CompletableFuture<ProfilingInfo> requestProfiling(ResourceID taskManagerId,
                                                         int duration,
                                                         ProfilingInfo.ProfilingMode mode,
                                                         java.time.Duration timeout)

Description copied from interface: ResourceManagerGateway

Requests the profiling instance from the given TaskExecutor.

Specified by:

requestProfiling in interface ResourceManagerGateway

Parameters:

taskManagerId - taskManagerId identifying the TaskExecutor to get the profiling from

duration - profiling duration

mode - profiling mode ProfilingInfo.ProfilingMode

timeout - timeout of the asynchronous operation

Returns:

Future containing the created profiling information

requestTaskExecutorThreadInfoGateway

public CompletableFuture<TaskExecutorThreadInfoGateway> requestTaskExecutorThreadInfoGateway(ResourceID taskManagerId,
                                                                                             org.apache.flink.api.common.time.Time timeout)

Description copied from interface: ResourceManagerGateway

Requests the TaskExecutorGateway.

Specified by:

requestTaskExecutorThreadInfoGateway in interface ResourceManagerGateway

Parameters:

taskManagerId - identifying the TaskExecutor.

Returns:

Future containing the task executor gateway.

notifyNewBlockedNodes

public CompletableFuture<Acknowledge> notifyNewBlockedNodes(Collection<BlockedNode> newNodes)

Description copied from interface: BlocklistListener

Notify new blocked node records.

Specified by:

notifyNewBlockedNodes in interface BlocklistListener

Parameters:

newNodes - the new blocked node records

Returns:

Future acknowledge once the new nodes have successfully notified.

registerMetrics

protected void registerMetrics()

closeJobManagerConnection

protected void closeJobManagerConnection(org.apache.flink.api.common.JobID jobId,
                                         org.apache.flink.runtime.resourcemanager.ResourceManager.ResourceRequirementHandling resourceRequirementHandling,
                                         Exception cause)

This method should be called by the framework once it detects that a currently registered job manager has failed.

Parameters:

jobId - identifying the job whose leader shall be disconnected.

resourceRequirementHandling - indicating how existing resource requirements for the corresponding job should be handled

cause - The exception which cause the JobManager failed.

closeTaskManagerConnection

protected Optional<WorkerType> closeTaskManagerConnection(ResourceID resourceID,
                                                          Exception cause)

This method should be called by the framework once it detects that a currently registered task executor has failed.

Parameters:

resourceID - Id of the TaskManager that has failed.

cause - The exception which cause the TaskManager failed.

Returns:

The WorkerType of the closed connection, or empty if already removed.

removeJob

protected void removeJob(org.apache.flink.api.common.JobID jobId,
                         Exception cause)

jobLeaderLostLeadership

protected void jobLeaderLostLeadership(org.apache.flink.api.common.JobID jobId,
                                       JobMasterId oldJobMasterId)

getInstanceIdByResourceId

@VisibleForTesting
public Optional<InstanceID> getInstanceIdByResourceId(ResourceID resourceID)

getWorkerByInstanceId

protected WorkerType getWorkerByInstanceId(InstanceID instanceId)

onFatalError

protected void onFatalError(Throwable t)

Notifies the ResourceManager that a fatal error has occurred and it cannot proceed.

Parameters:

t - The exception describing the fatal error

initialize

protected abstract void initialize()
                            throws ResourceManagerException

Initializes the framework specific components.

Throws:

ResourceManagerException - which occurs during initialization and causes the resource manager to fail.

terminate

protected abstract void terminate()
                           throws Exception

Terminates the framework specific components.

Throws:

Exception - which occurs during termination.

internalDeregisterApplication

protected abstract void internalDeregisterApplication(ApplicationStatus finalStatus,
                                                      @Nullable
                                                      String optionalDiagnostics)
                                               throws ResourceManagerException

The framework specific code to deregister the application. This should report the application's final status and shut down the resource manager cleanly.

This method also needs to make sure all pending containers that are not registered yet are returned.

Parameters:

finalStatus - The application status to report.

optionalDiagnostics - A diagnostics message or null.

Throws:

ResourceManagerException - if the application could not be shut down.

getWorkerNodeIfAcceptRegistration

protected abstract Optional<WorkerType> getWorkerNodeIfAcceptRegistration(ResourceID resourceID)

Get worker node if the worker resource is accepted.

Parameters:

resourceID - The worker resource id

stopWorkerIfSupported

public void stopWorkerIfSupported(WorkerType worker)

Stops the given worker if supported.

Parameters:

worker - The worker.

getReadyToServeFuture

protected abstract CompletableFuture<Void> getReadyToServeFuture()

Get the ready to serve future of the resource manager.

Returns:

The ready to serve future of the resource manager, which indicated whether it is ready to serve.

getResourceAllocator

protected abstract ResourceAllocator getResourceAllocator()

setFailUnfulfillableRequest

protected void setFailUnfulfillableRequest(boolean failUnfulfillableRequest)

Set SlotManager whether to fail unfulfillable slot requests.

Parameters:

failUnfulfillableRequest - whether to fail unfulfillable requests

onNewTokensObtained

public void onNewTokensObtained(byte[] tokens)
                         throws Exception

Description copied from interface: DelegationTokenManager.Listener

Callback function when new delegation tokens obtained.

Specified by:

onNewTokensObtained in interface DelegationTokenManager.Listener

Throws:

Exception