org.apache.hadoop.hbase.master.balancer

Class StochasticLoadBalancer

java.lang.Object

org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer

org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer

All Implemented Interfaces:

ConfigurationObserver, LoadBalancer, Stoppable

Direct Known Subclasses:

FavoredStochasticBalancer

@InterfaceAudience.LimitedPrivate(value="Configuration")
public class StochasticLoadBalancer
extends BaseLoadBalancer

This is a best effort load balancer. Given a Cost function F(C) => x It will randomly try and mutate the cluster to Cprime. If F(Cprime) < F(C) then the new cluster state becomes the plan. It includes costs functions to compute the cost of:

• Region Load
• Table Load
• Data Locality
• Memstore Sizes
• Storefile Sizes

Every cost function returns a number between 0 and 1 inclusive; where 0 is the lowest cost best solution, and 1 is the highest possible cost and the worst solution. The computed costs are scaled by their respective multipliers:

• hbase.master.balancer.stochastic.regionLoadCost
• hbase.master.balancer.stochastic.moveCost
• hbase.master.balancer.stochastic.tableLoadCost
• hbase.master.balancer.stochastic.localityCost
• hbase.master.balancer.stochastic.memstoreSizeCost
• hbase.master.balancer.stochastic.storefileSizeCost

You can also add custom Cost function by setting the the following configuration value:

• hbase.master.balancer.stochastic.additionalCostFunctions

All custom Cost Functions needs to extends CostFunction

In addition to the above configurations, the balancer can be tuned by the following configuration values:

• hbase.master.balancer.stochastic.maxMoveRegions which controls what the max number of regions that can be moved in a single invocation of this balancer.
• hbase.master.balancer.stochastic.stepsPerRegion is the coefficient by which the number of regions is multiplied to try and get the number of times the balancer will mutate all servers.
• hbase.master.balancer.stochastic.maxSteps which controls the maximum number of times that the balancer will try and mutate all the servers. The balancer will use the minimum of this value and the above computation.

This balancer is best used with hbase.master.loadbalance.bytable set to false so that the balancer gets the full picture of all loads on the cluster.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	StochasticLoadBalancer.GeneratorType

Field Summary

Fields

Modifier and Type	Field and Description
protected List<org.apache.hadoop.hbase.master.balancer.CandidateGenerator>	candidateGenerators
protected static String	COST_FUNCTIONS_COST_FUNCTIONS_KEY
protected static int	DEFAULT_KEEP_REGION_LOADS
protected static long	DEFAULT_MAX_RUNNING_TIME
protected static int	DEFAULT_MAX_STEPS
protected static float	DEFAULT_MIN_COST_NEED_BALANCE
protected static boolean	DEFAULT_RUN_MAX_STEPS
protected static int	DEFAULT_STEPS_PER_REGION
protected static String	KEEP_REGION_LOADS
protected static String	MAX_RUNNING_TIME_KEY
protected static String	MAX_STEPS_KEY
protected static String	MIN_COST_NEED_BALANCE_KEY
static String	OVERALL_COST_FUNCTION_NAME
protected static String	RUN_MAX_STEPS_KEY
protected static String	STEPS_PER_REGION_KEY

Fields inherited from class org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer

BALANCER_DECISION_BUFFER_ENABLED, BALANCER_REJECTION_BUFFER_ENABLED, clusterStatus, DEFAULT_BALANCER_DECISION_BUFFER_ENABLED, DEFAULT_BALANCER_REJECTION_BUFFER_ENABLED, DEFAULT_HBASE_MASTER_LOADBALANCE_BYTABLE, isByTable, masterServerName, metricsBalancer, MIN_SERVER_BALANCE, provider, rackManager, regionFinder, slop, useRegionFinder

Fields inherited from interface org.apache.hadoop.hbase.master.LoadBalancer

BOGUS_SERVER_NAME, HBASE_RSGROUP_LOADBALANCER_CLASS

Constructor Summary

Constructors

Constructor and Description
StochasticLoadBalancer() The constructor that pass a MetricsStochasticBalancer to BaseLoadBalancer to replace its default MetricsBalancer
StochasticLoadBalancer(MetricsStochasticBalancer metricsStochasticBalancer)

Method Summary

Modifier and Type	Method and Description
protected List<RegionPlan>	balanceTable(TableName tableName, Map<ServerName,List<RegionInfo>> loadOfOneTable) Given the cluster state this will try and approach an optimal balance.
protected List<org.apache.hadoop.hbase.master.balancer.CandidateGenerator>	createCandidateGenerators()
protected org.apache.hadoop.hbase.master.balancer.CandidateGenerator	getRandomGenerator() Select the candidate generator to use based on the cost of cost functions.
protected void	loadConf(org.apache.hadoop.conf.Configuration conf)
void	updateBalancerLoadInfo(Map<TableName,Map<ServerName,List<RegionInfo>>> loadOfAllTable) In some scenarios, Balancer needs to update internal status or information according to the current tables load
void	updateClusterMetrics(ClusterMetrics st) Set the current cluster status.

Methods inherited from class org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer

balanceCluster, getConf, getDefaultSlop, idleRegionServerExist, initialize, isStopped, onConfigurationChange, postMasterStartupInitialize, preBalanceCluster, randomAssignment, regionOffline, regionOnline, retainAssignment, roundRobinAssignment, setClusterInfoProvider, sloppyRegionServerExist, stop, toEnsumbleTableLoad, updateBalancerStatus

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

STEPS_PER_REGION_KEY

protected static final String STEPS_PER_REGION_KEY

See Also:

Constant Field Values

DEFAULT_STEPS_PER_REGION

protected static final int DEFAULT_STEPS_PER_REGION

See Also:

Constant Field Values

MAX_STEPS_KEY

protected static final String MAX_STEPS_KEY

See Also:

Constant Field Values

DEFAULT_MAX_STEPS

protected static final int DEFAULT_MAX_STEPS

See Also:

Constant Field Values

RUN_MAX_STEPS_KEY

protected static final String RUN_MAX_STEPS_KEY

See Also:

Constant Field Values

DEFAULT_RUN_MAX_STEPS

protected static final boolean DEFAULT_RUN_MAX_STEPS

See Also:

Constant Field Values

MAX_RUNNING_TIME_KEY

protected static final String MAX_RUNNING_TIME_KEY

See Also:

Constant Field Values

DEFAULT_MAX_RUNNING_TIME

protected static final long DEFAULT_MAX_RUNNING_TIME

See Also:

Constant Field Values

KEEP_REGION_LOADS

protected static final String KEEP_REGION_LOADS

See Also:

Constant Field Values

DEFAULT_KEEP_REGION_LOADS

protected static final int DEFAULT_KEEP_REGION_LOADS

See Also:

Constant Field Values

MIN_COST_NEED_BALANCE_KEY

protected static final String MIN_COST_NEED_BALANCE_KEY

See Also:

Constant Field Values

DEFAULT_MIN_COST_NEED_BALANCE

protected static final float DEFAULT_MIN_COST_NEED_BALANCE

See Also:

Constant Field Values

COST_FUNCTIONS_COST_FUNCTIONS_KEY

protected static final String COST_FUNCTIONS_COST_FUNCTIONS_KEY

See Also:

Constant Field Values

OVERALL_COST_FUNCTION_NAME

public static final String OVERALL_COST_FUNCTION_NAME

See Also:

Constant Field Values

candidateGenerators

protected List<org.apache.hadoop.hbase.master.balancer.CandidateGenerator> candidateGenerators

Constructor Detail

StochasticLoadBalancer

public StochasticLoadBalancer()

The constructor that pass a MetricsStochasticBalancer to BaseLoadBalancer to replace its default MetricsBalancer

StochasticLoadBalancer

public StochasticLoadBalancer(MetricsStochasticBalancer metricsStochasticBalancer)

Method Detail

createCandidateGenerators

protected List<org.apache.hadoop.hbase.master.balancer.CandidateGenerator> createCandidateGenerators()

loadConf

protected void loadConf(org.apache.hadoop.conf.Configuration conf)

Overrides:

loadConf in class BaseLoadBalancer

updateClusterMetrics

public void updateClusterMetrics(ClusterMetrics st)

Description copied from interface: LoadBalancer

Set the current cluster status. This allows a LoadBalancer to map host name to a server

Specified by:

updateClusterMetrics in interface LoadBalancer

Overrides:

updateClusterMetrics in class BaseLoadBalancer

updateBalancerLoadInfo

public void updateBalancerLoadInfo(Map<TableName,Map<ServerName,List<RegionInfo>>> loadOfAllTable)

Description copied from interface: LoadBalancer

In some scenarios, Balancer needs to update internal status or information according to the current tables load

Parameters:

loadOfAllTable - region load of servers for all table

getRandomGenerator

protected org.apache.hadoop.hbase.master.balancer.CandidateGenerator getRandomGenerator()

Select the candidate generator to use based on the cost of cost functions. The chance of selecting a candidate generator is propotional to the share of cost of all cost functions among all cost functions that benefit from it.

balanceTable

protected List<RegionPlan> balanceTable(TableName tableName,
                                        Map<ServerName,List<RegionInfo>> loadOfOneTable)

Given the cluster state this will try and approach an optimal balance. This should always approach the optimal state given enough steps.

Specified by:

balanceTable in class BaseLoadBalancer

Parameters:

tableName - the table to be balanced

loadOfOneTable - region load of servers for the specific one table

Returns:

List of plans