org.apache.hadoop.yarn.server.resourcemanager.reservation

Class InMemoryPlan

java.lang.Object

org.apache.hadoop.yarn.server.resourcemanager.reservation.InMemoryPlan

All Implemented Interfaces:

Plan, PlanContext, PlanEdit, PlanView

public class InMemoryPlan
extends Object
implements Plan

This class represents an in memory representation of the state of our reservation system, and provides accelerated access to both individual reservations and aggregate utilization of resources over time.

Constructor Summary

Constructors

Constructor and Description
InMemoryPlan(QueueMetrics queueMetrics, SharingPolicy policy, ReservationAgent agent, org.apache.hadoop.yarn.api.records.Resource totalCapacity, long step, org.apache.hadoop.yarn.util.resource.ResourceCalculator resCalc, org.apache.hadoop.yarn.api.records.Resource minAlloc, org.apache.hadoop.yarn.api.records.Resource maxAlloc, String queueName, Planner replanner, boolean getMoveOnExpiry, RMContext rmContext)
InMemoryPlan(QueueMetrics queueMetrics, SharingPolicy policy, ReservationAgent agent, org.apache.hadoop.yarn.api.records.Resource totalCapacity, long step, org.apache.hadoop.yarn.util.resource.ResourceCalculator resCalc, org.apache.hadoop.yarn.api.records.Resource minAlloc, org.apache.hadoop.yarn.api.records.Resource maxAlloc, String queueName, Planner replanner, boolean getMoveOnExpiry, RMContext rmContext, org.apache.hadoop.yarn.util.Clock clock)

Method Summary

Modifier and Type	Method and Description
boolean	addReservation(ReservationAllocation reservation, boolean isRecovering) Add a new ReservationAllocation to the plan
void	archiveCompletedReservations(long tick) Method invoked to garbage collect old reservations.
boolean	deleteReservation(org.apache.hadoop.yarn.api.records.ReservationId reservationID) Delete an existing ReservationAllocation from the plan identified uniquely by its ReservationId.
Set<ReservationAllocation>	getAllReservations() Gets all the reservations in the plan
RLESparseResourceAllocation	getAvailableResourceOverTime(String user, org.apache.hadoop.yarn.api.records.ReservationId oldId, long start, long end) This method returns the amount of resources available to a given user (optionally if removing a certain reservation) over the start-end time range.
RLESparseResourceAllocation	getConsumptionForUserOverTime(String user, long start, long end) This method returns a RLE encoded view of the user reservation utilization between start and end time.
long	getEarliestStartTime() Gets the time (UTC in ms) at which the first reservation starts
long	getLastEndTime() Returns the time (UTC in ms) at which the last reservation terminates
org.apache.hadoop.yarn.api.records.Resource	getMaximumAllocation() Returns the single largest Resource allocation that can be reserved in this plan
org.apache.hadoop.yarn.api.records.Resource	getMinimumAllocation() Returns the single smallest Resource allocation that can be reserved in this plan
boolean	getMoveOnExpiry() Instructs the PlanFollower on what to do for applications which are still running when the reservation is expiring (move-to-default vs kill)
QueueMetrics	getQueueMetrics() Return the QueueMetrics for the queue in the ResourceScheduler corresponding to this plan
String	getQueueName() Return the name of the queue in the ResourceScheduler corresponding to this plan
Planner	getReplanner() Return an instance of a Planner, which will be invoked in response to unexpected reduction in the resources of this plan
ReservationAgent	getReservationAgent() Return the ReservationAgent configured for this plan that is responsible for optimally placing various reservation requests
ReservationAllocation	getReservationById(org.apache.hadoop.yarn.api.records.ReservationId reservationID) Return a ReservationAllocation identified by its ReservationId
Set<ReservationAllocation>	getReservationByUserAtTime(String user, long t) Return a set of ReservationAllocation that belongs to a certain user and overlaps time t.
RLESparseResourceAllocation	getReservationCountForUserOverTime(String user, long start, long end) This method returns a RLE encoded view of the user reservation count utilization between start and end time.
Set<ReservationAllocation>	getReservations(org.apache.hadoop.yarn.api.records.ReservationId reservationID, ReservationInterval interval) Return a set of ReservationAllocation identified by any user.
Set<ReservationAllocation>	getReservations(org.apache.hadoop.yarn.api.records.ReservationId reservationID, ReservationInterval interval, String user) Return a set of ReservationAllocation identified by the user who made the reservation.
Set<ReservationAllocation>	getReservationsAtTime(long tick) Gets all the active reservations at the specified point of time
org.apache.hadoop.yarn.util.resource.ResourceCalculator	getResourceCalculator() Returns the system ResourceCalculator
SharingPolicy	getSharingPolicy() Return the configured SharingPolicy that governs the sharing of the resources of the plan between its various users
long	getStep() Returns the configured "step" or granularity of time of the plan in millis.
org.apache.hadoop.yarn.api.records.Resource	getTotalCapacity() Returns the overall capacity in terms of Resource assigned to this plan (typically will correspond to the absolute capacity of the corresponding queue).
org.apache.hadoop.yarn.api.records.Resource	getTotalCommittedResources(long t) Returns the total Resource reserved for all users at the specified time
void	setTotalCapacity(org.apache.hadoop.yarn.api.records.Resource cap) Sets the overall capacity in terms of Resource assigned to this plan
String	toCumulativeString()
String	toString()
boolean	updateReservation(ReservationAllocation reservation) Updates an existing ReservationAllocation in the plan.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

InMemoryPlan

public InMemoryPlan(QueueMetrics queueMetrics,
                    SharingPolicy policy,
                    ReservationAgent agent,
                    org.apache.hadoop.yarn.api.records.Resource totalCapacity,
                    long step,
                    org.apache.hadoop.yarn.util.resource.ResourceCalculator resCalc,
                    org.apache.hadoop.yarn.api.records.Resource minAlloc,
                    org.apache.hadoop.yarn.api.records.Resource maxAlloc,
                    String queueName,
                    Planner replanner,
                    boolean getMoveOnExpiry,
                    RMContext rmContext)

InMemoryPlan

public InMemoryPlan(QueueMetrics queueMetrics,
                    SharingPolicy policy,
                    ReservationAgent agent,
                    org.apache.hadoop.yarn.api.records.Resource totalCapacity,
                    long step,
                    org.apache.hadoop.yarn.util.resource.ResourceCalculator resCalc,
                    org.apache.hadoop.yarn.api.records.Resource minAlloc,
                    org.apache.hadoop.yarn.api.records.Resource maxAlloc,
                    String queueName,
                    Planner replanner,
                    boolean getMoveOnExpiry,
                    RMContext rmContext,
                    org.apache.hadoop.yarn.util.Clock clock)

Method Detail

getQueueMetrics

public QueueMetrics getQueueMetrics()

Description copied from interface: PlanContext

Return the QueueMetrics for the queue in the ResourceScheduler corresponding to this plan

Specified by:

getQueueMetrics in interface PlanContext

Returns:

the QueueMetrics for the queue in the ResourceScheduler corresponding to this plan

getAllReservations

public Set<ReservationAllocation> getAllReservations()

Description copied from interface: PlanView

Gets all the reservations in the plan

Specified by:

getAllReservations in interface PlanView

Returns:

set of all reservations handled by this Plan

addReservation

public boolean addReservation(ReservationAllocation reservation,
                              boolean isRecovering)
                       throws PlanningException

Description copied from interface: PlanEdit

Add a new ReservationAllocation to the plan

Specified by:

addReservation in interface PlanEdit

Parameters:

reservation - the ReservationAllocation to be added to the plan

isRecovering - flag to indicate if reservation is being added as part of failover or not

Returns:

true if addition is successful, false otherwise

Throws:

PlanningException

updateReservation

public boolean updateReservation(ReservationAllocation reservation)
                          throws PlanningException

Description copied from interface: PlanEdit

Updates an existing ReservationAllocation in the plan. This is required for re-negotiation

Specified by:

updateReservation in interface PlanEdit

Parameters:

reservation - the ReservationAllocation to be updated the plan

Returns:

true if update is successful, false otherwise

Throws:

PlanningException

deleteReservation

public boolean deleteReservation(org.apache.hadoop.yarn.api.records.ReservationId reservationID)

Description copied from interface: PlanEdit

Delete an existing ReservationAllocation from the plan identified uniquely by its ReservationId. This will generally be used for garbage collection

Specified by:

deleteReservation in interface PlanEdit

Parameters:

reservationID - the ReservationAllocation to be deleted from the plan identified uniquely by its ReservationId

Returns:

true if delete is successful, false otherwise

archiveCompletedReservations

public void archiveCompletedReservations(long tick)

Description copied from interface: PlanEdit

Method invoked to garbage collect old reservations. It cleans up expired reservations that have fallen out of the sliding archival window

Specified by:

archiveCompletedReservations in interface PlanEdit

Parameters:

tick - the current time from which the archival window is computed

getReservationsAtTime

public Set<ReservationAllocation> getReservationsAtTime(long tick)

Description copied from interface: PlanView

Gets all the active reservations at the specified point of time

Specified by:

getReservationsAtTime in interface PlanView

Parameters:

tick - the time (UTC in ms) for which the active reservations are requested

Returns:

set of active reservations at the specified time

getStep

public long getStep()

Description copied from interface: PlanContext

Returns the configured "step" or granularity of time of the plan in millis.

Specified by:

getStep in interface PlanContext

Returns:

plan step in millis

getSharingPolicy

public SharingPolicy getSharingPolicy()

Description copied from interface: PlanContext

Return the configured SharingPolicy that governs the sharing of the resources of the plan between its various users

Specified by:

getSharingPolicy in interface PlanContext

Returns:

the configured SharingPolicy that governs the sharing of the resources of the plan between its various users

getReservationAgent

public ReservationAgent getReservationAgent()

Description copied from interface: PlanContext

Return the ReservationAgent configured for this plan that is responsible for optimally placing various reservation requests

Specified by:

getReservationAgent in interface PlanContext

Returns:

the ReservationAgent configured for this plan

getReservationCountForUserOverTime

public RLESparseResourceAllocation getReservationCountForUserOverTime(String user,
                                                                      long start,
                                                                      long end)

Description copied from interface: PlanView

This method returns a RLE encoded view of the user reservation count utilization between start and end time.

Specified by:

getReservationCountForUserOverTime in interface PlanView

Returns:

RLE encoded view of reservation used over time

getConsumptionForUserOverTime

public RLESparseResourceAllocation getConsumptionForUserOverTime(String user,
                                                                 long start,
                                                                 long end)

Description copied from interface: PlanView

This method returns a RLE encoded view of the user reservation utilization between start and end time.

Specified by:

getConsumptionForUserOverTime in interface PlanView

Returns:

RLE encoded view of resources used over time

getTotalCommittedResources

public org.apache.hadoop.yarn.api.records.Resource getTotalCommittedResources(long t)

Description copied from interface: PlanView

Returns the total Resource reserved for all users at the specified time

Specified by:

getTotalCommittedResources in interface PlanView

Parameters:

t - the time (UTC in ms) for which the reserved resources are requested

Returns:

the total Resource reserved for all users at the specified time

getReservations

public Set<ReservationAllocation> getReservations(org.apache.hadoop.yarn.api.records.ReservationId reservationID,
                                                  ReservationInterval interval)

Description copied from interface: PlanView

Return a set of ReservationAllocation identified by any user.

Specified by:

getReservations in interface PlanView

Parameters:

reservationID - the unqiue id to identify the ReservationAllocation

interval - the time interval used to retrieve the reservation allocations from. Only reservations with start time no greater than the interval end time, and end time no less than the interval start time will be selected.

Returns:

a set of ReservationAllocation identified by any user

getReservations

public Set<ReservationAllocation> getReservations(org.apache.hadoop.yarn.api.records.ReservationId reservationID,
                                                  ReservationInterval interval,
                                                  String user)

Description copied from interface: PlanView

Return a set of ReservationAllocation identified by the user who made the reservation.

Specified by:

getReservations in interface PlanView

Parameters:

reservationID - the unqiue id to identify the ReservationAllocation

interval - the time interval used to retrieve the reservation allocations from. Only reservations with start time no greater than the interval end time, and end time no less than the interval start time will be selected.

user - the user to retrieve the reservation allocation from.

Returns:

a set of ReservationAllocation identified by the user who made the reservation

getReservationById

public ReservationAllocation getReservationById(org.apache.hadoop.yarn.api.records.ReservationId reservationID)

Description copied from interface: PlanView

Return a ReservationAllocation identified by its ReservationId

Specified by:

getReservationById in interface PlanView

Parameters:

reservationID - the unique id to identify the ReservationAllocation

Returns:

ReservationAllocation identified by the specified id

getTotalCapacity

public org.apache.hadoop.yarn.api.records.Resource getTotalCapacity()

Description copied from interface: PlanView

Returns the overall capacity in terms of Resource assigned to this plan (typically will correspond to the absolute capacity of the corresponding queue).

Specified by:

getTotalCapacity in interface PlanView

Returns:

the overall capacity in terms of Resource assigned to this plan

getAvailableResourceOverTime

public RLESparseResourceAllocation getAvailableResourceOverTime(String user,
                                                                org.apache.hadoop.yarn.api.records.ReservationId oldId,
                                                                long start,
                                                                long end)
                                                         throws PlanningException

Description copied from interface: PlanView

This method returns the amount of resources available to a given user (optionally if removing a certain reservation) over the start-end time range.

Specified by:

getAvailableResourceOverTime in interface PlanView

Returns:

a view of the plan as it is available to this user

Throws:

PlanningException

getMinimumAllocation

public org.apache.hadoop.yarn.api.records.Resource getMinimumAllocation()

Description copied from interface: PlanContext

Returns the single smallest Resource allocation that can be reserved in this plan

Specified by:

getMinimumAllocation in interface PlanContext

Returns:

the single smallest Resource allocation that can be reserved in this plan

setTotalCapacity

public void setTotalCapacity(org.apache.hadoop.yarn.api.records.Resource cap)

Description copied from interface: PlanEdit

Sets the overall capacity in terms of Resource assigned to this plan

Specified by:

setTotalCapacity in interface PlanEdit

Parameters:

cap - the overall capacity in terms of Resource assigned to this plan

getEarliestStartTime

public long getEarliestStartTime()

Description copied from interface: PlanView

Gets the time (UTC in ms) at which the first reservation starts

Specified by:

getEarliestStartTime in interface PlanView

Returns:

the time (UTC in ms) at which the first reservation starts

getLastEndTime

public long getLastEndTime()

Description copied from interface: PlanView

Returns the time (UTC in ms) at which the last reservation terminates

Specified by:

getLastEndTime in interface PlanView

Returns:

the time (UTC in ms) at which the last reservation terminates

getResourceCalculator

public org.apache.hadoop.yarn.util.resource.ResourceCalculator getResourceCalculator()

Description copied from interface: PlanContext

Returns the system ResourceCalculator

Specified by:

getResourceCalculator in interface PlanContext

Returns:

the system ResourceCalculator

getQueueName

public String getQueueName()

Description copied from interface: PlanContext

Return the name of the queue in the ResourceScheduler corresponding to this plan

Specified by:

getQueueName in interface PlanContext

Returns:

the name of the queue in the ResourceScheduler corresponding to this plan

getMaximumAllocation

public org.apache.hadoop.yarn.api.records.Resource getMaximumAllocation()

Description copied from interface: PlanContext

Returns the single largest Resource allocation that can be reserved in this plan

Specified by:

getMaximumAllocation in interface PlanContext

Returns:

the single largest Resource allocation that can be reserved in this plan

toCumulativeString

public String toCumulativeString()

getReplanner

public Planner getReplanner()

Description copied from interface: PlanContext

Return an instance of a Planner, which will be invoked in response to unexpected reduction in the resources of this plan

Specified by:

getReplanner in interface PlanContext

Returns:

an instance of a Planner, which will be invoked in response to unexpected reduction in the resources of this plan

getMoveOnExpiry

public boolean getMoveOnExpiry()

Description copied from interface: PlanContext

Instructs the PlanFollower on what to do for applications which are still running when the reservation is expiring (move-to-default vs kill)

Specified by:

getMoveOnExpiry in interface PlanContext

Returns:

true if remaining applications have to be killed, false if they have to migrated

toString

public String toString()

Overrides:

toString in class Object

getReservationByUserAtTime

public Set<ReservationAllocation> getReservationByUserAtTime(String user,
                                                             long t)

Description copied from interface: PlanView

Return a set of ReservationAllocation that belongs to a certain user and overlaps time t.

Specified by:

getReservationByUserAtTime in interface PlanView

Parameters:

user - the user being considered

t - the instant in time being considered

Returns:

set of active ReservationAllocations for this user at this time