org.apache.hadoop.yarn.server.resourcemanager.scheduler

Class AbstractYarnScheduler<T extends SchedulerApplicationAttempt,N extends SchedulerNode>

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler<T,N>

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.service.Service, org.apache.hadoop.yarn.event.EventHandler<SchedulerEvent>, Recoverable, ResourceScheduler, YarnScheduler

Direct Known Subclasses:

CapacityScheduler, FairScheduler, FifoScheduler

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public abstract class AbstractYarnScheduler<T extends SchedulerApplicationAttempt,N extends SchedulerNode>
extends org.apache.hadoop.service.AbstractService
implements ResourceScheduler

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Field Summary

Fields

Modifier and Type	Field and Description
protected ConcurrentMap<org.apache.hadoop.yarn.api.records.ApplicationId,SchedulerApplication<T>>	applications
protected org.apache.hadoop.yarn.api.records.Resource	clusterResource
protected static Allocation	EMPTY_ALLOCATION
protected static List<org.apache.hadoop.yarn.api.records.Container>	EMPTY_CONTAINER_LIST
protected org.apache.hadoop.yarn.api.records.Resource	maximumAllocation
protected org.apache.hadoop.yarn.api.records.Resource	minimumAllocation
protected int	nmExpireInterval
protected Map<org.apache.hadoop.yarn.api.records.NodeId,N>	nodes
protected RMContext	rmContext

Constructor Summary

Constructors

Constructor and Description
AbstractYarnScheduler(String name) Construct the service.

Method Summary

Modifier and Type	Method and Description
void	addBlacklistedNodeIdsToList(SchedulerApplicationAttempt app, List<org.apache.hadoop.yarn.api.records.NodeId> blacklistNodeIdList) Add blacklisted NodeIds to the list that is passed.
void	addQueue(Queue newQueue) Add to the scheduler a new Queue.
org.apache.hadoop.yarn.api.records.Priority	checkAndGetApplicationPriority(org.apache.hadoop.yarn.api.records.Priority priorityFromContext, String user, String queueName, org.apache.hadoop.yarn.api.records.ApplicationId applicationId) Verify whether a submitted application priority is valid as per configured Queue
void	clearPendingContainerCache()
void	completedContainer(RMContainer rmContainer, org.apache.hadoop.yarn.api.records.ContainerStatus containerStatus, RMContainerEventType event)
protected abstract void	completedContainerInternal(RMContainer rmContainer, org.apache.hadoop.yarn.api.records.ContainerStatus containerStatus, RMContainerEventType event)
protected void	containerIncreasedOnNode(org.apache.hadoop.yarn.api.records.ContainerId containerId, SchedulerNode node, org.apache.hadoop.yarn.api.records.Container increasedContainerReportedByNM)
protected void	containerLaunchedOnNode(org.apache.hadoop.yarn.api.records.ContainerId containerId, SchedulerNode node)
protected void	createReleaseCache()
protected List<SchedContainerChangeRequest>	createSchedContainerChangeRequests(List<org.apache.hadoop.yarn.api.records.UpdateContainerRequest> changeRequests, boolean increase)
protected abstract void	decreaseContainer(SchedContainerChangeRequest decreaseRequest, SchedulerApplicationAttempt attempt)
protected void	decreaseContainers(List<org.apache.hadoop.yarn.api.records.UpdateContainerRequest> decreaseRequests, SchedulerApplicationAttempt attempt)
T	getApplicationAttempt(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId)
org.apache.hadoop.yarn.api.records.ApplicationResourceUsageReport	getAppResourceUsageReport(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId) Get a resource usage report from a given app attempt ID.
org.apache.hadoop.yarn.api.records.Resource	getClusterResource() Get the whole resource capacity of the cluster.
T	getCurrentAttemptForContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)
org.apache.hadoop.yarn.api.records.Priority	getMaxClusterLevelAppPriority() Get cluster max priority.
org.apache.hadoop.yarn.api.records.Resource	getMaximumResourceCapability() Get maximum allocatable Resource at the cluster level.
org.apache.hadoop.yarn.api.records.Resource	getMaximumResourceCapability(String queueName) Get maximum allocatable Resource for the queue specified.
org.apache.hadoop.yarn.api.records.Resource	getMinimumResourceCapability() Get minimum allocatable Resource.
SchedulerNodeReport	getNodeReport(org.apache.hadoop.yarn.api.records.NodeId nodeId) Get node resource usage report.
List<org.apache.hadoop.yarn.api.records.ResourceRequest>	getPendingResourceRequestsForAttempt(org.apache.hadoop.yarn.api.records.ApplicationAttemptId attemptId)
Set<String>	getPlanQueues() Gets the list of names for queues managed by the Reservation System
RMContainer	getRMContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId) Get the container for the given containerId.
SchedulerAppReport	getSchedulerAppInfo(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId) Get the Scheduler app for a given app attempt Id.
Map<org.apache.hadoop.yarn.api.records.ApplicationId,SchedulerApplication<T>>	getSchedulerApplications()
SchedulerNode	getSchedulerNode(org.apache.hadoop.yarn.api.records.NodeId nodeId) Get SchedulerNode corresponds to nodeId.
EnumSet<org.apache.hadoop.yarn.proto.YarnServiceProtos.SchedulerResourceTypes>	getSchedulingResourceTypes() Return a collection of the resource types that are considered when scheduling
List<org.apache.hadoop.yarn.api.records.Container>	getTransferredContainers(org.apache.hadoop.yarn.api.records.ApplicationAttemptId currentAttempt) Get previous attempts' live containers for work-preserving AM restart.
protected void	initMaximumResourceCapability(org.apache.hadoop.yarn.api.records.Resource maximumAllocation)
void	killAllAppsInQueue(String queueName) Terminate all applications in the specified queue.
void	moveAllApps(String sourceQueue, String destQueue) Completely drain sourceQueue of applications, by moving all of them to destQueue.
String	moveApplication(org.apache.hadoop.yarn.api.records.ApplicationId appId, String newQueue) Moves the given application to the given queue
void	recoverContainersOnNode(List<org.apache.hadoop.yarn.server.api.protocolrecords.NMContainerStatus> containerReports, RMNode nm)
protected void	refreshMaximumAllocation(org.apache.hadoop.yarn.api.records.Resource newMaxAlloc)
protected void	releaseContainers(List<org.apache.hadoop.yarn.api.records.ContainerId> containers, SchedulerApplicationAttempt attempt)
void	removeQueue(String queueName) Remove an existing queue.
void	serviceInit(org.apache.hadoop.conf.Configuration conf)
void	setClusterMaxPriority(org.apache.hadoop.conf.Configuration conf) Set the cluster max priority
void	setEntitlement(String queue, QueueEntitlement entitlement) This method increase the entitlement for current queue (must respect invariants, e.g., no overcommit of parents, non negative, etc.).
void	updateApplicationPriority(org.apache.hadoop.yarn.api.records.Priority newPriority, org.apache.hadoop.yarn.api.records.ApplicationId applicationId) Change application priority of a submitted application at runtime
protected void	updateMaximumAllocation(SchedulerNode node, boolean add)
void	updateNodeResource(RMNode nm, org.apache.hadoop.yarn.api.records.ResourceOption resourceOption) Process resource update on a node.

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, serviceStart, serviceStop, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceScheduler

reinitialize, setRMContext

Methods inherited from interface org.apache.hadoop.yarn.server.resourcemanager.scheduler.YarnScheduler

allocate, checkAccess, getAppsInQueue, getNumClusterNodes, getQueueInfo, getQueueUserAclInfo, getResourceCalculator, getRootQueueMetrics

Methods inherited from interface org.apache.hadoop.yarn.event.EventHandler

handle

Methods inherited from interface org.apache.hadoop.yarn.server.resourcemanager.recovery.Recoverable

recover

Field Detail

nodes

protected Map<org.apache.hadoop.yarn.api.records.NodeId,N extends SchedulerNode> nodes

clusterResource

protected org.apache.hadoop.yarn.api.records.Resource clusterResource

minimumAllocation

protected org.apache.hadoop.yarn.api.records.Resource minimumAllocation

maximumAllocation

protected org.apache.hadoop.yarn.api.records.Resource maximumAllocation

rmContext

protected RMContext rmContext

applications

protected ConcurrentMap<org.apache.hadoop.yarn.api.records.ApplicationId,SchedulerApplication<T extends SchedulerApplicationAttempt>> applications

nmExpireInterval

protected int nmExpireInterval

EMPTY_CONTAINER_LIST

protected static final List<org.apache.hadoop.yarn.api.records.Container> EMPTY_CONTAINER_LIST

EMPTY_ALLOCATION

protected static final Allocation EMPTY_ALLOCATION

Constructor Detail

AbstractYarnScheduler

public AbstractYarnScheduler(String name)

Construct the service.

Parameters:

name - service name

Method Detail

serviceInit

public void serviceInit(org.apache.hadoop.conf.Configuration conf)
                 throws Exception

Overrides:

serviceInit in class org.apache.hadoop.service.AbstractService

Throws:

Exception

getTransferredContainers

public List<org.apache.hadoop.yarn.api.records.Container> getTransferredContainers(org.apache.hadoop.yarn.api.records.ApplicationAttemptId currentAttempt)

Description copied from interface: YarnScheduler

Get previous attempts' live containers for work-preserving AM restart.

Specified by:

getTransferredContainers in interface YarnScheduler

Parameters:

currentAttempt - the id of the application attempt

Returns:

list of live containers for the given attempt

getSchedulerApplications

public Map<org.apache.hadoop.yarn.api.records.ApplicationId,SchedulerApplication<T>> getSchedulerApplications()

addBlacklistedNodeIdsToList

public void addBlacklistedNodeIdsToList(SchedulerApplicationAttempt app,
                                        List<org.apache.hadoop.yarn.api.records.NodeId> blacklistNodeIdList)

Add blacklisted NodeIds to the list that is passed.

Parameters:

app - application attempt.

blacklistNodeIdList - the list to store blacklisted NodeIds.

getClusterResource

public org.apache.hadoop.yarn.api.records.Resource getClusterResource()

Description copied from interface: YarnScheduler

Get the whole resource capacity of the cluster.

Specified by:

getClusterResource in interface YarnScheduler

Returns:

the whole resource capacity of the cluster.

getMinimumResourceCapability

public org.apache.hadoop.yarn.api.records.Resource getMinimumResourceCapability()

Description copied from interface: YarnScheduler

Get minimum allocatable Resource.

Specified by:

getMinimumResourceCapability in interface YarnScheduler

Returns:

minimum allocatable resource

getMaximumResourceCapability

public org.apache.hadoop.yarn.api.records.Resource getMaximumResourceCapability()

Description copied from interface: YarnScheduler

Get maximum allocatable Resource at the cluster level.

Specified by:

getMaximumResourceCapability in interface YarnScheduler

Returns:

maximum allocatable resource

getMaximumResourceCapability

public org.apache.hadoop.yarn.api.records.Resource getMaximumResourceCapability(String queueName)

Description copied from interface: YarnScheduler

Get maximum allocatable Resource for the queue specified.

Specified by:

getMaximumResourceCapability in interface YarnScheduler

Parameters:

queueName - queue name

Returns:

maximum allocatable resource

initMaximumResourceCapability

protected void initMaximumResourceCapability(org.apache.hadoop.yarn.api.records.Resource maximumAllocation)

containerLaunchedOnNode

protected void containerLaunchedOnNode(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                                       SchedulerNode node)

containerIncreasedOnNode

protected void containerIncreasedOnNode(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                                        SchedulerNode node,
                                        org.apache.hadoop.yarn.api.records.Container increasedContainerReportedByNM)

getApplicationAttempt

public T getApplicationAttempt(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId)

getSchedulerAppInfo

public SchedulerAppReport getSchedulerAppInfo(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId)

Description copied from interface: YarnScheduler

Get the Scheduler app for a given app attempt Id.

Specified by:

getSchedulerAppInfo in interface YarnScheduler

Parameters:

appAttemptId - the id of the application attempt

Returns:

SchedulerApp for this given attempt.

getAppResourceUsageReport

public org.apache.hadoop.yarn.api.records.ApplicationResourceUsageReport getAppResourceUsageReport(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId)

Description copied from interface: YarnScheduler

Get a resource usage report from a given app attempt ID.

Specified by:

getAppResourceUsageReport in interface YarnScheduler

Parameters:

appAttemptId - the id of the application attempt

Returns:

resource usage report for this given attempt

getCurrentAttemptForContainer

public T getCurrentAttemptForContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)

getRMContainer

public RMContainer getRMContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)

Description copied from interface: YarnScheduler

Get the container for the given containerId.

Specified by:

getRMContainer in interface YarnScheduler

Returns:

the container for the given containerId.

getNodeReport

public SchedulerNodeReport getNodeReport(org.apache.hadoop.yarn.api.records.NodeId nodeId)

Description copied from interface: YarnScheduler

Get node resource usage report.

Specified by:

getNodeReport in interface YarnScheduler

Returns:

the SchedulerNodeReport for the node or null if nodeId does not point to a defined node.

moveApplication

public String moveApplication(org.apache.hadoop.yarn.api.records.ApplicationId appId,
                              String newQueue)
                       throws org.apache.hadoop.yarn.exceptions.YarnException

Description copied from interface: YarnScheduler

Moves the given application to the given queue

Specified by:

moveApplication in interface YarnScheduler

Returns:

the name of the queue the application was placed into

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - if the move cannot be carried out

removeQueue

public void removeQueue(String queueName)
                 throws org.apache.hadoop.yarn.exceptions.YarnException

Description copied from interface: YarnScheduler

Remove an existing queue. Implementations might limit when a queue could be removed (e.g., must have zero entitlement, and no applications running, or must be a leaf, etc..).

Specified by:

removeQueue in interface YarnScheduler

Parameters:

queueName - name of the queue to remove

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

addQueue

public void addQueue(Queue newQueue)
              throws org.apache.hadoop.yarn.exceptions.YarnException

Description copied from interface: YarnScheduler

Add to the scheduler a new Queue. Implementations might limit what type of queues can be dynamically added (e.g., Queue must be a leaf, must be attached to existing parent, must have zero entitlement).

Specified by:

addQueue in interface YarnScheduler

Parameters:

newQueue - the queue being added.

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

setEntitlement

public void setEntitlement(String queue,
                           QueueEntitlement entitlement)
                    throws org.apache.hadoop.yarn.exceptions.YarnException

Description copied from interface: YarnScheduler

This method increase the entitlement for current queue (must respect invariants, e.g., no overcommit of parents, non negative, etc.). Entitlement is a general term for weights in FairScheduler, capacity for the CapacityScheduler, etc.

Specified by:

setEntitlement in interface YarnScheduler

Parameters:

queue - the queue for which we change entitlement

entitlement - the new entitlement for the queue (capacity, maxCapacity, etc..)

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

recoverContainersOnNode

public void recoverContainersOnNode(List<org.apache.hadoop.yarn.server.api.protocolrecords.NMContainerStatus> containerReports,
                                    RMNode nm)

createReleaseCache

protected void createReleaseCache()

clearPendingContainerCache

public void clearPendingContainerCache()

completedContainer

@InterfaceAudience.Private
public void completedContainer(RMContainer rmContainer,
                                                          org.apache.hadoop.yarn.api.records.ContainerStatus containerStatus,
                                                          RMContainerEventType event)

completedContainerInternal

protected abstract void completedContainerInternal(RMContainer rmContainer,
                                                   org.apache.hadoop.yarn.api.records.ContainerStatus containerStatus,
                                                   RMContainerEventType event)

releaseContainers

protected void releaseContainers(List<org.apache.hadoop.yarn.api.records.ContainerId> containers,
                                 SchedulerApplicationAttempt attempt)

decreaseContainers

protected void decreaseContainers(List<org.apache.hadoop.yarn.api.records.UpdateContainerRequest> decreaseRequests,
                                  SchedulerApplicationAttempt attempt)

decreaseContainer

protected abstract void decreaseContainer(SchedContainerChangeRequest decreaseRequest,
                                          SchedulerApplicationAttempt attempt)

getSchedulerNode

public SchedulerNode getSchedulerNode(org.apache.hadoop.yarn.api.records.NodeId nodeId)

Description copied from interface: YarnScheduler

Get SchedulerNode corresponds to nodeId.

Specified by:

getSchedulerNode in interface YarnScheduler

Parameters:

nodeId - the node id of RMNode

Returns:

SchedulerNode corresponds to nodeId

moveAllApps

public void moveAllApps(String sourceQueue,
                        String destQueue)
                 throws org.apache.hadoop.yarn.exceptions.YarnException

Description copied from interface: YarnScheduler

Completely drain sourceQueue of applications, by moving all of them to destQueue.

Specified by:

moveAllApps in interface YarnScheduler

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

killAllAppsInQueue

public void killAllAppsInQueue(String queueName)
                        throws org.apache.hadoop.yarn.exceptions.YarnException

Description copied from interface: YarnScheduler

Terminate all applications in the specified queue.

Specified by:

killAllAppsInQueue in interface YarnScheduler

Parameters:

queueName - the name of queue to be drained

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

updateNodeResource

public void updateNodeResource(RMNode nm,
                               org.apache.hadoop.yarn.api.records.ResourceOption resourceOption)

Process resource update on a node.

getSchedulingResourceTypes

public EnumSet<org.apache.hadoop.yarn.proto.YarnServiceProtos.SchedulerResourceTypes> getSchedulingResourceTypes()

Return a collection of the resource types that are considered when scheduling

Specified by:

getSchedulingResourceTypes in interface YarnScheduler

Returns:

an EnumSet containing the resource types

getPlanQueues

public Set<String> getPlanQueues()
                          throws org.apache.hadoop.yarn.exceptions.YarnException

Description copied from interface: YarnScheduler

Gets the list of names for queues managed by the Reservation System

Specified by:

getPlanQueues in interface YarnScheduler

Returns:

the list of queues which support reservations

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

updateMaximumAllocation

protected void updateMaximumAllocation(SchedulerNode node,
                                       boolean add)

refreshMaximumAllocation

protected void refreshMaximumAllocation(org.apache.hadoop.yarn.api.records.Resource newMaxAlloc)

getPendingResourceRequestsForAttempt

public List<org.apache.hadoop.yarn.api.records.ResourceRequest> getPendingResourceRequestsForAttempt(org.apache.hadoop.yarn.api.records.ApplicationAttemptId attemptId)

Specified by:

getPendingResourceRequestsForAttempt in interface YarnScheduler

checkAndGetApplicationPriority

public org.apache.hadoop.yarn.api.records.Priority checkAndGetApplicationPriority(org.apache.hadoop.yarn.api.records.Priority priorityFromContext,
                                                                                  String user,
                                                                                  String queueName,
                                                                                  org.apache.hadoop.yarn.api.records.ApplicationId applicationId)
                                                                           throws org.apache.hadoop.yarn.exceptions.YarnException

Description copied from interface: YarnScheduler

Verify whether a submitted application priority is valid as per configured Queue

Specified by:

checkAndGetApplicationPriority in interface YarnScheduler

Parameters:

priorityFromContext - Submitted Application priority.

user - User who submitted the Application

queueName - Name of the Queue

applicationId - Application ID

Returns:

Updated Priority from scheduler

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

updateApplicationPriority

public void updateApplicationPriority(org.apache.hadoop.yarn.api.records.Priority newPriority,
                                      org.apache.hadoop.yarn.api.records.ApplicationId applicationId)
                               throws org.apache.hadoop.yarn.exceptions.YarnException

Description copied from interface: YarnScheduler

Change application priority of a submitted application at runtime

Specified by:

updateApplicationPriority in interface YarnScheduler

Parameters:

newPriority - Submitted Application priority.

applicationId - Application ID

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

getMaxClusterLevelAppPriority

public org.apache.hadoop.yarn.api.records.Priority getMaxClusterLevelAppPriority()

Description copied from interface: YarnScheduler

Get cluster max priority.

Specified by:

getMaxClusterLevelAppPriority in interface YarnScheduler

Returns:

maximum priority of cluster

setClusterMaxPriority

public void setClusterMaxPriority(org.apache.hadoop.conf.Configuration conf)
                           throws org.apache.hadoop.yarn.exceptions.YarnException

Description copied from interface: YarnScheduler

Set the cluster max priority

Specified by:

setClusterMaxPriority in interface YarnScheduler

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

createSchedContainerChangeRequests

protected List<SchedContainerChangeRequest> createSchedContainerChangeRequests(List<org.apache.hadoop.yarn.api.records.UpdateContainerRequest> changeRequests,
                                                                               boolean increase)