org.apache.hadoop.yarn.server.resourcemanager.reservation

Class AbstractSchedulerPlanFollower

java.lang.Object

org.apache.hadoop.yarn.server.resourcemanager.reservation.AbstractSchedulerPlanFollower

All Implemented Interfaces:

Runnable, PlanFollower

Direct Known Subclasses:

CapacitySchedulerPlanFollower, FairSchedulerPlanFollower

public abstract class AbstractSchedulerPlanFollower
extends Object
implements PlanFollower

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.yarn.util.Clock	clock
protected Collection<Plan>	plans
protected YarnScheduler	scheduler

Constructor Summary

Constructors

Constructor and Description
AbstractSchedulerPlanFollower()

Method Summary

Modifier and Type	Method and Description
protected abstract void	addReservationQueue(String planQueueName, Queue queue, String currResId) Add a new reservation queue for reservation currResId for this planQueue
protected void	cleanupExpiredQueues(String planQueueName, boolean shouldMove, Set<String> toRemove, String defReservationQueue) First sets entitlement of queues to zero to prevent new app submission.
protected abstract void	createDefaultReservationQueue(String planQueueName, Queue queue, String defReservationQueue) Creates the default reservation queue for use when no reservation is used for applications submitted to this planQueue
protected abstract List<? extends Queue>	getChildReservationQueues(Queue planQueue) Get a list of reservation queues for this planQueue
protected abstract Queue	getPlanQueue(String planQueueName) Get queue associated with reservable queue named
protected abstract org.apache.hadoop.yarn.api.records.Resource	getPlanResources(Plan plan, Queue queue, org.apache.hadoop.yarn.api.records.Resource clusterResources) Get plan resources for this planQueue
protected String	getReservationIdFromQueueName(String resQueueName)
protected String	getReservationQueueName(String planQueueName, String reservationId)
protected abstract org.apache.hadoop.yarn.api.records.Resource	getReservationQueueResourceIfExists(Plan plan, org.apache.hadoop.yarn.api.records.ReservationId reservationId) Get reservation queue resources if it exists otherwise return null.
protected int	getReservedResources(long now, Set<ReservationAllocation> currentReservations, Set<String> curReservationNames, org.apache.hadoop.yarn.api.records.Resource reservedResources)
void	init(org.apache.hadoop.yarn.util.Clock clock, ResourceScheduler sched, Collection<Plan> plans) Init function that configures the PlanFollower, by providing:
void	run()
void	setPlans(Collection<Plan> plans) Setter for the list of plans.
protected void	setQueueEntitlement(String planQueueName, String currResId, float targetCapacity, float maxCapacity)
protected List<ReservationAllocation>	sortByDelta(List<ReservationAllocation> currentReservations, long now, Plan plan) Sort in the order from the least new amount of resources asked (likely negative) to the highest.
void	synchronizePlan(Plan plan, boolean shouldReplan) The function performing the actual synchronization operation for a given Plan.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

plans

protected Collection<Plan> plans

scheduler

protected YarnScheduler scheduler

clock

protected org.apache.hadoop.yarn.util.Clock clock

Constructor Detail

AbstractSchedulerPlanFollower

public AbstractSchedulerPlanFollower()

Method Detail

init

public void init(org.apache.hadoop.yarn.util.Clock clock,
                 ResourceScheduler sched,
                 Collection<Plan> plans)

Description copied from interface: PlanFollower

Init function that configures the PlanFollower, by providing:

Specified by:

init in interface PlanFollower

Parameters:

clock - a reference to the system clock.

sched - a reference to the underlying scheduler

plans - references to the plans we should keep synchronized at every time tick.

run

public void run()

Specified by:

run in interface Runnable

setPlans

public void setPlans(Collection<Plan> plans)

Description copied from interface: PlanFollower

Setter for the list of plans.

Specified by:

setPlans in interface PlanFollower

Parameters:

plans - the collection of Plans we operate on at every time tick.

synchronizePlan

public void synchronizePlan(Plan plan,
                            boolean shouldReplan)

Description copied from interface: PlanFollower

The function performing the actual synchronization operation for a given Plan. This is normally invoked by the run method, but it can be invoked synchronously to avoid race conditions when a user's reservation request start time is imminent.

Specified by:

synchronizePlan in interface PlanFollower

Parameters:

plan - the Plan to synchronize

shouldReplan - replan on reduction of plan capacity if true or proportionally scale down reservations if false

getReservationIdFromQueueName

protected String getReservationIdFromQueueName(String resQueueName)

setQueueEntitlement

protected void setQueueEntitlement(String planQueueName,
                                   String currResId,
                                   float targetCapacity,
                                   float maxCapacity)
                            throws org.apache.hadoop.yarn.exceptions.YarnException

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

getReservationQueueName

protected String getReservationQueueName(String planQueueName,
                                         String reservationId)

cleanupExpiredQueues

protected void cleanupExpiredQueues(String planQueueName,
                                    boolean shouldMove,
                                    Set<String> toRemove,
                                    String defReservationQueue)

First sets entitlement of queues to zero to prevent new app submission. Then move all apps in the set of queues to the parent plan queue's default reservation queue if move is enabled. Finally cleanups the queue by killing any apps (if move is disabled or move failed) and removing the queue

getReservedResources

protected int getReservedResources(long now,
                                   Set<ReservationAllocation> currentReservations,
                                   Set<String> curReservationNames,
                                   org.apache.hadoop.yarn.api.records.Resource reservedResources)

sortByDelta

protected List<ReservationAllocation> sortByDelta(List<ReservationAllocation> currentReservations,
                                                  long now,
                                                  Plan plan)

Sort in the order from the least new amount of resources asked (likely negative) to the highest. This prevents "order-of-operation" errors related to exceeding 100% capacity temporarily.

getPlanQueue

protected abstract Queue getPlanQueue(String planQueueName)

Get queue associated with reservable queue named

Parameters:

planQueueName - Name of the reservable queue

Returns:

queue associated with the reservable queue

getChildReservationQueues

protected abstract List<? extends Queue> getChildReservationQueues(Queue planQueue)

Get a list of reservation queues for this planQueue

addReservationQueue

protected abstract void addReservationQueue(String planQueueName,
                                            Queue queue,
                                            String currResId)

Add a new reservation queue for reservation currResId for this planQueue

createDefaultReservationQueue

protected abstract void createDefaultReservationQueue(String planQueueName,
                                                      Queue queue,
                                                      String defReservationQueue)

Creates the default reservation queue for use when no reservation is used for applications submitted to this planQueue

getPlanResources

protected abstract org.apache.hadoop.yarn.api.records.Resource getPlanResources(Plan plan,
                                                                                Queue queue,
                                                                                org.apache.hadoop.yarn.api.records.Resource clusterResources)

Get plan resources for this planQueue

getReservationQueueResourceIfExists

protected abstract org.apache.hadoop.yarn.api.records.Resource getReservationQueueResourceIfExists(Plan plan,
                                                                                                   org.apache.hadoop.yarn.api.records.ReservationId reservationId)

Get reservation queue resources if it exists otherwise return null.