org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair

Class FSAppAttempt

java.lang.Object

org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt

org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt

All Implemented Interfaces:

Schedulable, SchedulableEntity

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public class FSAppAttempt
extends SchedulerApplicationAttempt
implements Schedulable

Represents an application attempt from the viewpoint of the Fair Scheduler.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt

SchedulerApplicationAttempt.AMState

Field Summary

Fields inherited from class org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt

appAMNodePartitionName, appSchedulingInfo, attemptId, attemptResourceUsage, isStopped, lastMemoryAggregateAllocationUpdateTime, lastScheduledContainer, liveContainers, newlyAllocatedContainers, newlyDecreasedContainers, newlyIncreasedContainers, queue, reservedContainers, rmContext, updatedNMTokens

Constructor Summary

Constructors

Constructor and Description
FSAppAttempt(FairScheduler scheduler, org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId, String user, FSLeafQueue queue, ActiveUsersManager activeUsersManager, RMContext rmContext)

Method Summary

Modifier and Type	Method and Description
void	addPreemption(RMContainer container, long time)
RMContainer	allocate(NodeType type, FSSchedulerNode node, org.apache.hadoop.yarn.api.records.Priority priority, org.apache.hadoop.yarn.api.records.ResourceRequest request, org.apache.hadoop.yarn.api.records.Container container)
org.apache.hadoop.yarn.api.records.Resource	assignContainer(FSSchedulerNode node) Assign a container on this node if possible, and return the amount of resources assigned.
boolean	assignReservedContainer(FSSchedulerNode node) Called when this application already has an existing reservation on the given node.
void	clearPreemptedResources()
void	containerCompleted(RMContainer rmContainer, org.apache.hadoop.yarn.api.records.ContainerStatus containerStatus, RMContainerEventType event)
org.apache.hadoop.yarn.api.records.Container	createContainer(FSSchedulerNode node, org.apache.hadoop.yarn.api.records.Resource capability, org.apache.hadoop.yarn.api.records.Priority priority) Create and return a container object reflecting an allocation for the given appliction on the given node with the given capability and priority.
NodeType	getAllowedLocalityLevel(org.apache.hadoop.yarn.api.records.Priority priority, int numNodes, double nodeLocalityThreshold, double rackLocalityThreshold) Return the level at which we are allowed to schedule containers, given the current size of the cluster and thresholds indicating how many nodes to fail at (as a fraction of cluster size) before relaxing scheduling constraints.
NodeType	getAllowedLocalityLevelByTime(org.apache.hadoop.yarn.api.records.Priority priority, long nodeLocalityDelayMs, long rackLocalityDelayMs, long currentTimeMs) Return the level at which we are allowed to schedule containers.
Long	getContainerPreemptionTime(RMContainer container)
org.apache.hadoop.yarn.api.records.Resource	getDemand() Maximum number of resources required by this Schedulable.
org.apache.hadoop.yarn.api.records.Resource	getFairShare() Get the fair share assigned to this Schedulable.
org.apache.hadoop.yarn.api.records.Resource	getHeadroom() Headroom depends on resources in the cluster, current usage of the queue, queue's fair-share and queue's max-resources.
float	getLocalityWaitFactor(org.apache.hadoop.yarn.api.records.Priority priority, int clusterNodes)
org.apache.hadoop.yarn.api.records.Resource	getMaxShare() Maximum Resource share assigned to the schedulable.
QueueMetrics	getMetrics() Get metrics reference from containing queue.
org.apache.hadoop.yarn.api.records.Resource	getMinShare() Minimum Resource share assigned to the schedulable.
String	getName() Name of job/queue, used for debugging as well as for breaking ties in scheduling order deterministically.
org.apache.hadoop.yarn.api.records.Resource	getPreemptedResources()
Set<RMContainer>	getPreemptionContainers()
org.apache.hadoop.yarn.api.records.Priority	getPriority() Get the priority of the application
FSLeafQueue	getQueue()
org.apache.hadoop.yarn.api.records.Resource	getResourceUsage() Get the aggregate amount of resources consumed by the schedulable.
ResourceWeights	getResourceWeights()
long	getStartTime() Start time for jobs in FIFO queues; meaningless for QueueSchedulables.
ResourceWeights	getWeights() Job/queue weight in fair sharing.
RMContainer	preemptContainer() Preempt a running container according to the priority
void	resetAllowedLocalityLevel(org.apache.hadoop.yarn.api.records.Priority priority, NodeType level) Should be called when the scheduler assigns a container at a higher degree of locality than the current threshold.
void	resetPreemptedResources()
void	setFairShare(org.apache.hadoop.yarn.api.records.Resource fairShare) Assign a fair share to this Schedulable.
void	unreserve(org.apache.hadoop.yarn.api.records.Priority priority, FSSchedulerNode node) Remove the reservation on node at the given Priority.
void	updateDemand() Refresh the Schedulable's demand and those of its children if any.

Methods inherited from class org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt

addMissedNonPartitionedRequestSchedulingOpportunity, addReReservation, addSchedulingOpportunity, compareInputOrderTo, containerLaunchedOnNode, decreaseContainer, getActivedAppDiagnosticMessage, getAMResource, getAMResource, getAppAMNodePartitionName, getAppAttemptResourceUsage, getApplicationAttemptId, getApplicationId, getAppSchedulingInfo, getBlacklistedNodes, getCurrentConsumption, getCurrentReservation, getId, getLastScheduledContainer, getLiveContainers, getLiveContainersMap, getNewContainerId, getNumReservedContainers, getPendingAppDiagnosticMessage, getPendingRelease, getPriorities, getQueueName, getReReservations, getReservedContainers, getResource, getResourceLimit, getResourceRequest, getResourceRequests, getResourceUsageReport, getRMContainer, getSchedulingOpportunities, getSchedulingResourceUsage, getTotalRequiredResources, getUnmanagedAM, getUser, hasPendingResourceRequest, incNumAllocatedContainers, increaseContainer, isAmRunning, isPending, isPlaceBlacklisted, isRecovering, isReserved, isStopped, isWaitingForAMContainer, move, pullNewlyAllocatedContainers, pullNewlyDecreasedContainers, pullNewlyIncreasedContainers, pullUpdatedNMTokens, recordContainerAllocationTime, recordContainerRequestTime, recoverContainer, recoverResourceRequestsForContainer, removeIncreaseRequest, reserve, reserveIncreasedContainer, resetMissedNonPartitionedRequestSchedulingOpportunity, resetReReservations, resetSchedulingOpportunities, resetSchedulingOpportunities, setAMResource, setAMResource, setAmRunning, setAppAMNodePartitionName, setApplicationHeadroomForMetrics, setAttemptRecovering, setHeadroom, setPriority, showRequests, stop, subtractSchedulingOpportunity, transferStateFromPreviousAttempt, updateAMContainerDiagnostics, updateBlacklist, updateIncreaseRequests, updateResourceRequests

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FSAppAttempt

public FSAppAttempt(FairScheduler scheduler,
                    org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId,
                    String user,
                    FSLeafQueue queue,
                    ActiveUsersManager activeUsersManager,
                    RMContext rmContext)

Method Detail

getResourceWeights

public ResourceWeights getResourceWeights()

getMetrics

public QueueMetrics getMetrics()

Get metrics reference from containing queue.

containerCompleted

public void containerCompleted(RMContainer rmContainer,
                               org.apache.hadoop.yarn.api.records.ContainerStatus containerStatus,
                               RMContainerEventType event)

getHeadroom

public org.apache.hadoop.yarn.api.records.Resource getHeadroom()

Headroom depends on resources in the cluster, current usage of the queue, queue's fair-share and queue's max-resources.

Overrides:

getHeadroom in class SchedulerApplicationAttempt

Returns:

available resource headroom

getLocalityWaitFactor

public float getLocalityWaitFactor(org.apache.hadoop.yarn.api.records.Priority priority,
                                   int clusterNodes)

getAllowedLocalityLevel

public NodeType getAllowedLocalityLevel(org.apache.hadoop.yarn.api.records.Priority priority,
                                        int numNodes,
                                        double nodeLocalityThreshold,
                                        double rackLocalityThreshold)

Return the level at which we are allowed to schedule containers, given the current size of the cluster and thresholds indicating how many nodes to fail at (as a fraction of cluster size) before relaxing scheduling constraints.

getAllowedLocalityLevelByTime

public NodeType getAllowedLocalityLevelByTime(org.apache.hadoop.yarn.api.records.Priority priority,
                                              long nodeLocalityDelayMs,
                                              long rackLocalityDelayMs,
                                              long currentTimeMs)

Return the level at which we are allowed to schedule containers. Given the thresholds indicating how much time passed before relaxing scheduling constraints.

allocate

public RMContainer allocate(NodeType type,
                            FSSchedulerNode node,
                            org.apache.hadoop.yarn.api.records.Priority priority,
                            org.apache.hadoop.yarn.api.records.ResourceRequest request,
                            org.apache.hadoop.yarn.api.records.Container container)

resetAllowedLocalityLevel

public void resetAllowedLocalityLevel(org.apache.hadoop.yarn.api.records.Priority priority,
                                      NodeType level)

Should be called when the scheduler assigns a container at a higher degree of locality than the current threshold. Reset the allowed locality level to a higher degree of locality.

addPreemption

public void addPreemption(RMContainer container,
                          long time)

getContainerPreemptionTime

public Long getContainerPreemptionTime(RMContainer container)

getPreemptionContainers

public Set<RMContainer> getPreemptionContainers()

getQueue

public FSLeafQueue getQueue()

Overrides:

getQueue in class SchedulerApplicationAttempt

getPreemptedResources

public org.apache.hadoop.yarn.api.records.Resource getPreemptedResources()

resetPreemptedResources

public void resetPreemptedResources()

clearPreemptedResources

public void clearPreemptedResources()

createContainer

public org.apache.hadoop.yarn.api.records.Container createContainer(FSSchedulerNode node,
                                                                    org.apache.hadoop.yarn.api.records.Resource capability,
                                                                    org.apache.hadoop.yarn.api.records.Priority priority)

Create and return a container object reflecting an allocation for the given appliction on the given node with the given capability and priority.

unreserve

public void unreserve(org.apache.hadoop.yarn.api.records.Priority priority,
                      FSSchedulerNode node)

Remove the reservation on node at the given Priority. This dispatches SchedulerNode handlers as well.

assignReservedContainer

public boolean assignReservedContainer(FSSchedulerNode node)

Called when this application already has an existing reservation on the given node. Sees whether we can turn the reservation into an allocation. Also checks whether the application needs the reservation anymore, and releases it if not.

Parameters:

node - Node that the application has an existing reservation on

Returns:

whether the reservation on the given node is valid.

getName

public String getName()

Description copied from interface: Schedulable

Name of job/queue, used for debugging as well as for breaking ties in scheduling order deterministically.

Specified by:

getName in interface Schedulable

getDemand

public org.apache.hadoop.yarn.api.records.Resource getDemand()

Description copied from interface: Schedulable

Maximum number of resources required by this Schedulable. This is defined as number of currently utilized resources + number of unlaunched resources (that are either not yet launched or need to be speculated).

Specified by:

getDemand in interface Schedulable

getStartTime

public long getStartTime()

Description copied from interface: Schedulable

Start time for jobs in FIFO queues; meaningless for QueueSchedulables.

Specified by:

getStartTime in interface Schedulable

getMinShare

public org.apache.hadoop.yarn.api.records.Resource getMinShare()

Description copied from interface: Schedulable

Minimum Resource share assigned to the schedulable.

Specified by:

getMinShare in interface Schedulable

getMaxShare

public org.apache.hadoop.yarn.api.records.Resource getMaxShare()

Description copied from interface: Schedulable

Maximum Resource share assigned to the schedulable.

Specified by:

getMaxShare in interface Schedulable

getResourceUsage

public org.apache.hadoop.yarn.api.records.Resource getResourceUsage()

Description copied from interface: Schedulable

Get the aggregate amount of resources consumed by the schedulable.

Specified by:

getResourceUsage in interface Schedulable

getWeights

public ResourceWeights getWeights()

Description copied from interface: Schedulable

Job/queue weight in fair sharing.

Specified by:

getWeights in interface Schedulable

getPriority

public org.apache.hadoop.yarn.api.records.Priority getPriority()

Description copied from interface: SchedulableEntity

Get the priority of the application

Specified by:

getPriority in interface Schedulable

Specified by:

getPriority in interface SchedulableEntity

Overrides:

getPriority in class SchedulerApplicationAttempt

getFairShare

public org.apache.hadoop.yarn.api.records.Resource getFairShare()

Description copied from interface: Schedulable

Get the fair share assigned to this Schedulable.

Specified by:

getFairShare in interface Schedulable

setFairShare

public void setFairShare(org.apache.hadoop.yarn.api.records.Resource fairShare)

Description copied from interface: Schedulable

Assign a fair share to this Schedulable.

Specified by:

setFairShare in interface Schedulable

updateDemand

public void updateDemand()

Description copied from interface: Schedulable

Refresh the Schedulable's demand and those of its children if any.

Specified by:

updateDemand in interface Schedulable

assignContainer

public org.apache.hadoop.yarn.api.records.Resource assignContainer(FSSchedulerNode node)

Description copied from interface: Schedulable

Assign a container on this node if possible, and return the amount of resources assigned.

Specified by:

assignContainer in interface Schedulable

preemptContainer

public RMContainer preemptContainer()

Preempt a running container according to the priority

Specified by:

preemptContainer in interface Schedulable