org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity

Class LeafQueue

java.lang.Object

org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue

org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue

All Implemented Interfaces:

CSQueue, Queue

Direct Known Subclasses:

ReservationQueue

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public class LeafQueue
extends AbstractCSQueue

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	LeafQueue.User

Field Summary

Fields

Modifier and Type	Field and Description
protected int	maxApplications
protected int	maxApplicationsPerUser

Fields inherited from class org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue

authorizer, csContext, queueEntity

Constructor Summary

Constructors

Constructor and Description
LeafQueue(CapacitySchedulerContext cs, String queueName, CSQueue parent, CSQueue old)

Method Summary

Modifier and Type	Method and Description
CSAssignment	assignContainers(org.apache.hadoop.yarn.api.records.Resource clusterResource, FiCaSchedulerNode node, ResourceLimits currentResourceLimits, SchedulingMode schedulingMode) Assign containers to applications in the queue or it's children (if any).
void	attachContainer(org.apache.hadoop.yarn.api.records.Resource clusterResource, FiCaSchedulerApp application, RMContainer rmContainer) Attach a container to this queue
org.apache.hadoop.yarn.api.records.Resource	calculateAndGetAMResourceLimit()
org.apache.hadoop.yarn.api.records.Resource	calculateAndGetAMResourceLimitPerPartition(String nodePartition)
protected boolean	canAssignToUser(org.apache.hadoop.yarn.api.records.Resource clusterResource, String userName, org.apache.hadoop.yarn.api.records.Resource limit, FiCaSchedulerApp application, String nodePartition, ResourceLimits currentResourceLimits)
void	collectSchedulerApplications(Collection<org.apache.hadoop.yarn.api.records.ApplicationAttemptId> apps) Adds all applications in the queue and its subqueues to the given collection.
void	completedContainer(org.apache.hadoop.yarn.api.records.Resource clusterResource, FiCaSchedulerApp application, FiCaSchedulerNode node, RMContainer rmContainer, org.apache.hadoop.yarn.api.records.ContainerStatus containerStatus, RMContainerEventType event, CSQueue childQueue, boolean sortQueues) A container assigned to the queue has completed.
void	decAMUsedResource(String nodeLabel, org.apache.hadoop.yarn.api.records.Resource resourceToDec, SchedulerApplicationAttempt application)
void	decreaseContainer(org.apache.hadoop.yarn.api.records.Resource clusterResource, SchedContainerChangeRequest decreaseRequest, FiCaSchedulerApp app) Decrease container resource in the queue
void	decUsedResource(String nodeLabel, org.apache.hadoop.yarn.api.records.Resource resourceToDec, SchedulerApplicationAttempt application) When partition of node updated, we will update queue's resource usage if it has container(s) running on that.
void	detachContainer(org.apache.hadoop.yarn.api.records.Resource clusterResource, FiCaSchedulerApp application, RMContainer rmContainer) Detach a container from this queue
void	finishApplication(org.apache.hadoop.yarn.api.records.ApplicationId application, String user) An application submitted to this queue has finished.
void	finishApplicationAttempt(FiCaSchedulerApp application, String queue) An application attempt submitted to this queue has finished.
ActiveUsersManager	getActiveUsersManager() Get the ActiveUsersManager for the queue.
Collection<FiCaSchedulerApp>	getAllApplications() Obtain (read-only) collection of all applications.
org.apache.hadoop.yarn.api.records.Resource	getAMResourceLimit()
org.apache.hadoop.yarn.api.records.Resource	getAMResourceLimitPerPartition(String nodePartition)
Collection<FiCaSchedulerApp>	getApplications() Obtain (read-only) collection of active applications.
List<CSQueue>	getChildQueues() Get child queues
org.apache.hadoop.yarn.api.records.Priority	getDefaultApplicationPriority() Get the Default Application Priority for this queue
protected org.apache.hadoop.yarn.api.records.Resource	getHeadroom(LeafQueue.User user, org.apache.hadoop.yarn.api.records.Resource queueCurrentLimit, org.apache.hadoop.yarn.api.records.Resource clusterResource, FiCaSchedulerApp application)
protected org.apache.hadoop.yarn.api.records.Resource	getHeadroom(LeafQueue.User user, org.apache.hadoop.yarn.api.records.Resource queueCurrentLimit, org.apache.hadoop.yarn.api.records.Resource clusterResource, FiCaSchedulerApp application, String partition)
Map<String,TreeSet<RMContainer>>	getIgnoreExclusivityRMContainers()
float	getMaxAMResourcePerQueuePercent() Used only by tests.
int	getMaxApplications()
int	getMaxApplicationsPerUser()
float	getMinimumAllocationFactor() Used only by tests.
int	getNodeLocalityDelay()
int	getNumActiveApplications()
int	getNumActiveApplications(String user)
int	getNumApplications() Get the number of applications in the queue.
int	getNumApplications(String user)
int	getNumPendingApplications()
int	getNumPendingApplications(String user)
OrderingPolicy<FiCaSchedulerApp>	getOrderingPolicy()
Collection<FiCaSchedulerApp>	getPendingApplications() Obtain (read-only) collection of pending applications.
OrderingPolicy<FiCaSchedulerApp>	getPendingAppsOrderingPolicy()
org.apache.hadoop.yarn.api.records.QueueInfo	getQueueInfo(boolean includeChildQueues, boolean recursive) Get queue information
String	getQueuePath() Get the full name of the queue, including the heirarchy.
List<org.apache.hadoop.yarn.api.records.QueueUserACLInfo>	getQueueUserAclInfo(org.apache.hadoop.security.UserGroupInformation user) Get queue ACLs for given user.
boolean	getRackLocalityFullReset()
org.apache.hadoop.yarn.api.records.QueueState	getState() Get the current run-state of the queue
org.apache.hadoop.yarn.api.records.Resource	getTotalPendingResourcesConsideringUserLimit(org.apache.hadoop.yarn.api.records.Resource resources, String partition)
float	getUsageRatio(String label)
LeafQueue.User	getUser(String userName)
org.apache.hadoop.yarn.api.records.Resource	getUserAMResourceLimit()
org.apache.hadoop.yarn.api.records.Resource	getUserAMResourceLimitPerPartition(String nodePartition)
int	getUserLimit()
float	getUserLimitFactor()
org.apache.hadoop.yarn.api.records.Resource	getUserLimitPerUser(String userName, org.apache.hadoop.yarn.api.records.Resource resources, String partition)
ArrayList<UserInfo>	getUsers()
void	incAMUsedResource(String nodeLabel, org.apache.hadoop.yarn.api.records.Resource resourceToInc, SchedulerApplicationAttempt application)
void	incUsedResource(String nodeLabel, org.apache.hadoop.yarn.api.records.Resource resourceToInc, SchedulerApplicationAttempt application) When partition of node updated, we will update queue's resource usage if it has container(s) running on that.
void	recoverContainer(org.apache.hadoop.yarn.api.records.Resource clusterResource, SchedulerApplicationAttempt attempt, RMContainer rmContainer) Recover the state of the queue for a given container.
void	reinitialize(CSQueue newlyParsedQueue, org.apache.hadoop.yarn.api.records.Resource clusterResource) Reinitialize the queue.
void	removeApplicationAttempt(FiCaSchedulerApp application, LeafQueue.User user)
void	setAbsoluteCapacity(float absoluteCapacity)
void	setCapacity(float capacity)
void	setMaxApplications(int maxApplications)
void	setNodeLabelManager(RMNodeLabelsManager mgr)
void	setOrderingPolicy(OrderingPolicy<FiCaSchedulerApp> orderingPolicy)
protected void	setupQueueConfigs(org.apache.hadoop.yarn.api.records.Resource clusterResource)
void	submitApplication(org.apache.hadoop.yarn.api.records.ApplicationId applicationId, String userName, String queue) Submit a new application to the queue.
void	submitApplicationAttempt(FiCaSchedulerApp application, String userName) Submit an application attempt to the queue.
String	toString()
void	unreserveIncreasedContainer(org.apache.hadoop.yarn.api.records.Resource clusterResource, FiCaSchedulerApp app, FiCaSchedulerNode node, RMContainer rmContainer) We have a reserved increased container in the queue, we need to unreserve it.
void	updateClusterResource(org.apache.hadoop.yarn.api.records.Resource clusterResource, ResourceLimits currentResourceLimits) Update the cluster resource for queues as we add/remove nodes

Methods inherited from class org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue

accessibleToPartition, decPendingResource, decReservedResource, getAbsoluteCapacity, getAbsoluteMaximumCapacity, getAbsoluteUsedCapacity, getAccessibleNodeLabels, getACLs, getCapacity, getDefaultNodeLabelExpression, getKillableContainers, getMaximumAllocation, getMaximumCapacity, getMetrics, getMinimumAllocation, getNodeLabelsForQueue, getNumContainers, getParent, getPreemptionDisabled, getPrivilegedEntity, getQueueCapacities, getQueueInfo, getQueueName, getQueueResourceUsage, getQueueStatistics, getReservationContinueLooking, getTotalKillableResource, getUsedCapacity, getUsedResources, hasAccess, incPendingResource, incReservedResource, releaseResource, setAbsoluteUsedCapacity, setParent, setupConfigurableCapacities, setUsedCapacity

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

maxApplications

protected int maxApplications

maxApplicationsPerUser

protected int maxApplicationsPerUser

Constructor Detail

LeafQueue

public LeafQueue(CapacitySchedulerContext cs,
                 String queueName,
                 CSQueue parent,
                 CSQueue old)
          throws IOException

Throws:

IOException

Method Detail

setupQueueConfigs

protected void setupQueueConfigs(org.apache.hadoop.yarn.api.records.Resource clusterResource)
                          throws IOException

Throws:

IOException

getQueuePath

public String getQueuePath()

Description copied from interface: CSQueue

Get the full name of the queue, including the heirarchy.

Returns:

the full name of the queue

getMinimumAllocationFactor

@InterfaceAudience.Private
public float getMinimumAllocationFactor()

Used only by tests.

getMaxAMResourcePerQueuePercent

@InterfaceAudience.Private
public float getMaxAMResourcePerQueuePercent()

Used only by tests.

getMaxApplications

public int getMaxApplications()

getMaxApplicationsPerUser

public int getMaxApplicationsPerUser()

getActiveUsersManager

public ActiveUsersManager getActiveUsersManager()

Description copied from interface: CSQueue

Get the ActiveUsersManager for the queue.

Returns:

the ActiveUsersManager for the queue

getChildQueues

public List<CSQueue> getChildQueues()

Description copied from interface: CSQueue

Get child queues

Returns:

child queues

getNumApplications

public int getNumApplications()

Description copied from interface: CSQueue

Get the number of applications in the queue.

Returns:

number of applications

getNumPendingApplications

public int getNumPendingApplications()

getNumActiveApplications

public int getNumActiveApplications()

getNumApplications

@InterfaceAudience.Private
public int getNumApplications(String user)

getNumPendingApplications

@InterfaceAudience.Private
public int getNumPendingApplications(String user)

getNumActiveApplications

@InterfaceAudience.Private
public int getNumActiveApplications(String user)

getState

public org.apache.hadoop.yarn.api.records.QueueState getState()

Description copied from interface: CSQueue

Get the current run-state of the queue

Specified by:

getState in interface CSQueue

Overrides:

getState in class AbstractCSQueue

Returns:

current run-state

getUserLimit

@InterfaceAudience.Private
public int getUserLimit()

getUserLimitFactor

@InterfaceAudience.Private
public float getUserLimitFactor()

getQueueInfo

public org.apache.hadoop.yarn.api.records.QueueInfo getQueueInfo(boolean includeChildQueues,
                                                                 boolean recursive)

Description copied from interface: Queue

Get queue information

Parameters:

includeChildQueues - include child queues?

recursive - recursively get child queue information?

Returns:

queue information

getQueueUserAclInfo

public List<org.apache.hadoop.yarn.api.records.QueueUserACLInfo> getQueueUserAclInfo(org.apache.hadoop.security.UserGroupInformation user)

Description copied from interface: Queue

Get queue ACLs for given user.

Parameters:

user - username

Returns:

queue ACLs for user

toString

public String toString()

Overrides:

toString in class Object

setNodeLabelManager

public void setNodeLabelManager(RMNodeLabelsManager mgr)

getUser

public LeafQueue.User getUser(String userName)

getUsers

public ArrayList<UserInfo> getUsers()

Returns:

an ArrayList of UserInfo objects who are active in this queue

reinitialize

public void reinitialize(CSQueue newlyParsedQueue,
                         org.apache.hadoop.yarn.api.records.Resource clusterResource)
                  throws IOException

Description copied from interface: CSQueue

Reinitialize the queue.

Parameters:

newlyParsedQueue - new queue to re-initalize from

clusterResource - resources in the cluster

Throws:

IOException

submitApplicationAttempt

public void submitApplicationAttempt(FiCaSchedulerApp application,
                                     String userName)

Description copied from interface: CSQueue

Submit an application attempt to the queue.

submitApplication

public void submitApplication(org.apache.hadoop.yarn.api.records.ApplicationId applicationId,
                              String userName,
                              String queue)
                       throws org.apache.hadoop.security.AccessControlException

Description copied from interface: CSQueue

Submit a new application to the queue.

Parameters:

applicationId - the applicationId of the application being submitted

userName - user who submitted the application

queue - queue to which the application is submitted

Throws:

org.apache.hadoop.security.AccessControlException

getAMResourceLimit

public org.apache.hadoop.yarn.api.records.Resource getAMResourceLimit()

getAMResourceLimitPerPartition

public org.apache.hadoop.yarn.api.records.Resource getAMResourceLimitPerPartition(String nodePartition)

calculateAndGetAMResourceLimit

public org.apache.hadoop.yarn.api.records.Resource calculateAndGetAMResourceLimit()

getUserAMResourceLimit

public org.apache.hadoop.yarn.api.records.Resource getUserAMResourceLimit()

getUserAMResourceLimitPerPartition

public org.apache.hadoop.yarn.api.records.Resource getUserAMResourceLimitPerPartition(String nodePartition)

calculateAndGetAMResourceLimitPerPartition

public org.apache.hadoop.yarn.api.records.Resource calculateAndGetAMResourceLimitPerPartition(String nodePartition)

finishApplication

public void finishApplication(org.apache.hadoop.yarn.api.records.ApplicationId application,
                              String user)

Description copied from interface: CSQueue

An application submitted to this queue has finished.

user - user who submitted the application

finishApplicationAttempt

public void finishApplicationAttempt(FiCaSchedulerApp application,
                                     String queue)

Description copied from interface: CSQueue

An application attempt submitted to this queue has finished.

removeApplicationAttempt

public void removeApplicationAttempt(FiCaSchedulerApp application,
                                     LeafQueue.User user)

assignContainers

public CSAssignment assignContainers(org.apache.hadoop.yarn.api.records.Resource clusterResource,
                                     FiCaSchedulerNode node,
                                     ResourceLimits currentResourceLimits,
                                     SchedulingMode schedulingMode)

Description copied from interface: CSQueue

Assign containers to applications in the queue or it's children (if any).

Parameters:

clusterResource - the resource of the cluster.

node - node on which resources are available

currentResourceLimits - how much overall resource of this queue can use.

schedulingMode - Type of exclusive check when assign container on a NodeManager, see SchedulingMode.

Returns:

the assignment

getHeadroom

protected org.apache.hadoop.yarn.api.records.Resource getHeadroom(LeafQueue.User user,
                                                                  org.apache.hadoop.yarn.api.records.Resource queueCurrentLimit,
                                                                  org.apache.hadoop.yarn.api.records.Resource clusterResource,
                                                                  FiCaSchedulerApp application)

getHeadroom

protected org.apache.hadoop.yarn.api.records.Resource getHeadroom(LeafQueue.User user,
                                                                  org.apache.hadoop.yarn.api.records.Resource queueCurrentLimit,
                                                                  org.apache.hadoop.yarn.api.records.Resource clusterResource,
                                                                  FiCaSchedulerApp application,
                                                                  String partition)

getNodeLocalityDelay

@Lock(value=org.apache.hadoop.yarn.server.utils.Lock.NoLock.class)
public int getNodeLocalityDelay()

getRackLocalityFullReset

@Lock(value=org.apache.hadoop.yarn.server.utils.Lock.NoLock.class)
public boolean getRackLocalityFullReset()

canAssignToUser

@InterfaceAudience.Private
protected boolean canAssignToUser(org.apache.hadoop.yarn.api.records.Resource clusterResource,
                                                             String userName,
                                                             org.apache.hadoop.yarn.api.records.Resource limit,
                                                             FiCaSchedulerApp application,
                                                             String nodePartition,
                                                             ResourceLimits currentResourceLimits)

unreserveIncreasedContainer

public void unreserveIncreasedContainer(org.apache.hadoop.yarn.api.records.Resource clusterResource,
                                        FiCaSchedulerApp app,
                                        FiCaSchedulerNode node,
                                        RMContainer rmContainer)

Description copied from interface: CSQueue

We have a reserved increased container in the queue, we need to unreserve it. Since we just want to cancel the reserved increase request instead of stop the container, we shouldn't call completedContainer for such purpose.

completedContainer

public void completedContainer(org.apache.hadoop.yarn.api.records.Resource clusterResource,
                               FiCaSchedulerApp application,
                               FiCaSchedulerNode node,
                               RMContainer rmContainer,
                               org.apache.hadoop.yarn.api.records.ContainerStatus containerStatus,
                               RMContainerEventType event,
                               CSQueue childQueue,
                               boolean sortQueues)

Description copied from interface: CSQueue

A container assigned to the queue has completed.

Parameters:

clusterResource - the resource of the cluster

application - application to which the container was assigned

node - node on which the container completed

rmContainer - completed container, null if it was just a reservation

containerStatus - ContainerStatus for the completed container

event - event to be sent to the container

childQueue - CSQueue to reinsert in childQueues

sortQueues - indicates whether it should re-sort the queues

updateClusterResource

public void updateClusterResource(org.apache.hadoop.yarn.api.records.Resource clusterResource,
                                  ResourceLimits currentResourceLimits)

Description copied from interface: CSQueue

Update the cluster resource for queues as we add/remove nodes

Parameters:

clusterResource - the current cluster resource

currentResourceLimits - the current ResourceLimits

incUsedResource

public void incUsedResource(String nodeLabel,
                            org.apache.hadoop.yarn.api.records.Resource resourceToInc,
                            SchedulerApplicationAttempt application)

Description copied from interface: CSQueue

When partition of node updated, we will update queue's resource usage if it has container(s) running on that.

Specified by:

incUsedResource in interface CSQueue

Overrides:

incUsedResource in class AbstractCSQueue

decUsedResource

public void decUsedResource(String nodeLabel,
                            org.apache.hadoop.yarn.api.records.Resource resourceToDec,
                            SchedulerApplicationAttempt application)

Description copied from interface: CSQueue

When partition of node updated, we will update queue's resource usage if it has container(s) running on that.

Specified by:

decUsedResource in interface CSQueue

Overrides:

decUsedResource in class AbstractCSQueue

incAMUsedResource

public void incAMUsedResource(String nodeLabel,
                              org.apache.hadoop.yarn.api.records.Resource resourceToInc,
                              SchedulerApplicationAttempt application)

decAMUsedResource

public void decAMUsedResource(String nodeLabel,
                              org.apache.hadoop.yarn.api.records.Resource resourceToDec,
                              SchedulerApplicationAttempt application)

getUsageRatio

public float getUsageRatio(String label)

recoverContainer

public void recoverContainer(org.apache.hadoop.yarn.api.records.Resource clusterResource,
                             SchedulerApplicationAttempt attempt,
                             RMContainer rmContainer)

Description copied from interface: Queue

Recover the state of the queue for a given container.

Parameters:

clusterResource - the resource of the cluster

attempt - the application for which the container was allocated

rmContainer - the container that was recovered.

getPendingApplications

public Collection<FiCaSchedulerApp> getPendingApplications()

Obtain (read-only) collection of pending applications.

getApplications

public Collection<FiCaSchedulerApp> getApplications()

Obtain (read-only) collection of active applications.

getAllApplications

public Collection<FiCaSchedulerApp> getAllApplications()

Obtain (read-only) collection of all applications.

getTotalPendingResourcesConsideringUserLimit

public org.apache.hadoop.yarn.api.records.Resource getTotalPendingResourcesConsideringUserLimit(org.apache.hadoop.yarn.api.records.Resource resources,
                                                                                                String partition)

getUserLimitPerUser

public org.apache.hadoop.yarn.api.records.Resource getUserLimitPerUser(String userName,
                                                                       org.apache.hadoop.yarn.api.records.Resource resources,
                                                                       String partition)

collectSchedulerApplications

public void collectSchedulerApplications(Collection<org.apache.hadoop.yarn.api.records.ApplicationAttemptId> apps)

Description copied from interface: CSQueue

Adds all applications in the queue and its subqueues to the given collection.

Parameters:

apps - the collection to add the applications to

attachContainer

public void attachContainer(org.apache.hadoop.yarn.api.records.Resource clusterResource,
                            FiCaSchedulerApp application,
                            RMContainer rmContainer)

Description copied from interface: CSQueue

Attach a container to this queue

Parameters:

clusterResource - the current cluster resource

application - application to which the container was assigned

rmContainer - the container to attach

detachContainer

public void detachContainer(org.apache.hadoop.yarn.api.records.Resource clusterResource,
                            FiCaSchedulerApp application,
                            RMContainer rmContainer)

Description copied from interface: CSQueue

Detach a container from this queue

Parameters:

clusterResource - the current cluster resource

application - application to which the container was assigned

rmContainer - the container to detach

getIgnoreExclusivityRMContainers

public Map<String,TreeSet<RMContainer>> getIgnoreExclusivityRMContainers()

Returns:

all ignored partition exclusivity RMContainers in the LeafQueue, this will be used by preemption policy, and use of return ignorePartitionExclusivityRMContainer should protected by LeafQueue synchronized lock

setCapacity

public void setCapacity(float capacity)

setAbsoluteCapacity

public void setAbsoluteCapacity(float absoluteCapacity)

setMaxApplications

public void setMaxApplications(int maxApplications)

getOrderingPolicy

public OrderingPolicy<FiCaSchedulerApp> getOrderingPolicy()

setOrderingPolicy

public void setOrderingPolicy(OrderingPolicy<FiCaSchedulerApp> orderingPolicy)

getDefaultApplicationPriority

public org.apache.hadoop.yarn.api.records.Priority getDefaultApplicationPriority()

Description copied from interface: Queue

Get the Default Application Priority for this queue

Specified by:

getDefaultApplicationPriority in interface Queue

Overrides:

getDefaultApplicationPriority in class AbstractCSQueue

Returns:

default application priority

decreaseContainer

public void decreaseContainer(org.apache.hadoop.yarn.api.records.Resource clusterResource,
                              SchedContainerChangeRequest decreaseRequest,
                              FiCaSchedulerApp app)
                       throws org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException

Description copied from interface: CSQueue

Decrease container resource in the queue

Parameters:

clusterResource - Total cluster resource

decreaseRequest - The decrease request

app - The application of interest

Throws:

org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException

getPendingAppsOrderingPolicy

public OrderingPolicy<FiCaSchedulerApp> getPendingAppsOrderingPolicy()