org.apache.tez.runtime.library.resources
Class WeightedScalingMemoryDistributor

java.lang.Object
  org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.tez.runtime.common.resources.InitialMemoryAllocator

public class WeightedScalingMemoryDistributor
extends Object
implements org.apache.tez.runtime.common.resources.InitialMemoryAllocator

Distributes memory between various requesting components by applying a weighted scaling function. Overall, ensures that all requestors stay within the JVM limits. Configuration involves specifying weights for the different Inputs available in the tez-runtime-library. As an example, SortedShuffle : SortedOutput : UnsortedShuffle could be configured to be 20:10:1. In this case, if both SortedShuffle and UnsortedShuffle ask for the same amount of initial memory, SortedShuffle will be given 20 times more; both may be scaled down to fit within the JVM though.

Nested Class Summary

static class

WeightedScalingMemoryDistributor.RequestType

Constructor Summary

WeightedScalingMemoryDistributor()

Method Summary

Iterable<Long>	assignMemory(long availableForAllocation, int numTotalInputs, int numTotalOutputs, Iterable<org.apache.tez.runtime.common.resources.InitialMemoryRequestContext> initialRequests)
org.apache.hadoop.conf.Configuration	getConf()
void	setConf(org.apache.hadoop.conf.Configuration conf)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WeightedScalingMemoryDistributor

public WeightedScalingMemoryDistributor()

Method Detail

assignMemory

public Iterable<Long> assignMemory(long availableForAllocation,
                                   int numTotalInputs,
                                   int numTotalOutputs,
                                   Iterable<org.apache.tez.runtime.common.resources.InitialMemoryRequestContext> initialRequests)

Specified by:

assignMemory in interface org.apache.tez.runtime.common.resources.InitialMemoryAllocator

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable