org.apache.gobblin.util.request_allocation

Interface RequestAllocator<T extends Request<T>>

Type Parameters:

T -

All Known Implementing Classes:

BruteForceAllocator, GreedyAllocator, HierarchicalAllocator, PreOrderAllocator, PriorityIterableBasedRequestAllocator

public interface RequestAllocator<T extends Request<T>>

This interface is intended to solve the problem of selecting a subset of expensive requests based on priority when there are limited resources to perform such requests.

We assume there are a number of Requestors each one sending a finite stream of Requests. Each such request requires a certain amount of resources, and there is a finite pool of resources available. Additionally, some requests have higher priority than others. Our objective is to select a set of requests to satisfy such that their total resource usage is within the bounds of the finite resource pool, and such that, as much as possible, a request will not be selected if there was a request with a higher priority that was not selected.

We model the problem as follows:

1. A request is an implementation of Request.
2. A Requestor is a stream (Iterator) of Requests. We use a stream as opposed to a set or list because the Requestor is encouraged to lazily materialize requests only as needed by the request allocator.
3. A ResourcePool is a vector of doubles representing the available resources along a sequence of dimensions (e.g. bytes, files to copy).
4. A ResourceRequirement is a vector of doubles representing the resources need by a particular request. We assume that resource requirements are combined exclusively through vector addition.
5. A ResourceEstimator is a class that, given a Request, computes its ResourceRequirement.
6. A prioritizer is a Comparator which, given two Requests, determines which is higher priority (smaller is higher priority, following the PriorityQueue model).

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static interface	RequestAllocator.Factory

Method Summary

Modifier and Type	Method and Description
AllocatedRequestsIterator<T>	allocateRequests(Iterator<? extends Requestor<T>> requestors, ResourcePool resourcePool) Compute the subset of accepted Requests from the input Requestors which fit withing the ResourcePool.

Method Detail

allocateRequests

AllocatedRequestsIterator<T> allocateRequests(Iterator<? extends Requestor<T>> requestors,
                                              ResourcePool resourcePool)

Compute the subset of accepted Requests from the input Requestors which fit withing the ResourcePool.