org.apache.hadoop.hbase.master.balancer

Class FavoredStochasticBalancer

java.lang.Object

org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer

org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer

org.apache.hadoop.hbase.master.balancer.FavoredStochasticBalancer

All Implemented Interfaces:

ConfigurationObserver, FavoredNodesPromoter, LoadBalancer, Stoppable

@InterfaceAudience.Private
public class FavoredStochasticBalancer
extends StochasticLoadBalancer
implements FavoredNodesPromoter

An implementation of the LoadBalancer that assigns favored nodes for each region. There is a Primary RegionServer that hosts the region, and then there is Secondary and Tertiary RegionServers. Currently, the favored nodes information is used in creating HDFS files - the Primary RegionServer passes the primary, secondary, tertiary node addresses as hints to the DistributedFileSystem API for creating files on the filesystem. These nodes are treated as hints by the HDFS to place the blocks of the file. This alleviates the problem to do with reading from remote nodes (since we can make the Secondary RegionServer as the new Primary RegionServer) after a region is recovered. This should help provide consistent read latencies for the regions even when their primary region servers die. This provides two CandidateGenerator

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer

StochasticLoadBalancer.GeneratorType

Field Summary

Fields inherited from class org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer

candidateGenerators, COST_FUNCTIONS_COST_FUNCTIONS_KEY, DEFAULT_KEEP_REGION_LOADS, DEFAULT_MAX_RUNNING_TIME, DEFAULT_MAX_STEPS, DEFAULT_MIN_COST_NEED_BALANCE, DEFAULT_RUN_MAX_STEPS, DEFAULT_STEPS_PER_REGION, KEEP_REGION_LOADS, MAX_RUNNING_TIME_KEY, MAX_STEPS_KEY, MIN_COST_NEED_BALANCE_KEY, OVERALL_COST_FUNCTION_NAME, RUN_MAX_STEPS_KEY, STEPS_PER_REGION_KEY

Fields inherited from class org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer

BALANCER_DECISION_BUFFER_ENABLED, BALANCER_REJECTION_BUFFER_ENABLED, clusterStatus, DEFAULT_BALANCER_DECISION_BUFFER_ENABLED, DEFAULT_BALANCER_REJECTION_BUFFER_ENABLED, DEFAULT_HBASE_MASTER_LOADBALANCE_BYTABLE, isByTable, masterServerName, metricsBalancer, MIN_SERVER_BALANCE, provider, rackManager, regionFinder, slop, useRegionFinder

Fields inherited from interface org.apache.hadoop.hbase.favored.FavoredNodesPromoter

FAVORED_ALWAYS_ASSIGN_REGIONS

Fields inherited from interface org.apache.hadoop.hbase.master.LoadBalancer

BOGUS_SERVER_NAME, HBASE_RSGROUP_LOADBALANCER_CLASS

Constructor Summary

Constructors

Constructor and Description
FavoredStochasticBalancer()

Method Summary

Modifier and Type	Method and Description
protected List<RegionPlan>	balanceTable(TableName tableName, Map<ServerName,List<RegionInfo>> loadOfOneTable) For all regions correctly assigned to favored nodes, we just use the stochastic balancer implementation.
protected List<org.apache.hadoop.hbase.master.balancer.CandidateGenerator>	createCandidateGenerators()
void	generateFavoredNodesForDaughter(List<ServerName> servers, RegionInfo parent, RegionInfo regionA, RegionInfo regionB) Generate Favored Nodes for daughters during region split.
void	generateFavoredNodesForMergedRegion(RegionInfo merged, RegionInfo[] mergeParents) Generate favored nodes for a region during merge.
List<ServerName>	getFavoredNodes(RegionInfo regionInfo)
protected org.apache.hadoop.hbase.master.balancer.CandidateGenerator	getRandomGenerator() Returns any candidate generator in random
ServerName	randomAssignment(RegionInfo regionInfo, List<ServerName> servers) If we have favored nodes for a region, we will return one of the FN as destination.
Map<ServerName,List<RegionInfo>>	retainAssignment(Map<RegionInfo,ServerName> regions, List<ServerName> servers) Reuse BaseLoadBalancer's retainAssignment, but generate favored nodes when its missing.
Map<ServerName,List<RegionInfo>>	roundRobinAssignment(List<RegionInfo> regions, List<ServerName> servers) Round robin assignment: Segregate the regions into two types: 1.
void	setFavoredNodesManager(FavoredNodesManager fnm)

Methods inherited from class org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer

loadConf, updateBalancerLoadInfo, updateClusterMetrics

Methods inherited from class org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer

balanceCluster, getConf, getDefaultSlop, idleRegionServerExist, initialize, isStopped, onConfigurationChange, postMasterStartupInitialize, preBalanceCluster, regionOffline, regionOnline, setClusterInfoProvider, sloppyRegionServerExist, stop, toEnsumbleTableLoad, updateBalancerStatus

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FavoredStochasticBalancer

public FavoredStochasticBalancer()

Method Detail

setFavoredNodesManager

public void setFavoredNodesManager(FavoredNodesManager fnm)

Specified by:

setFavoredNodesManager in interface FavoredNodesPromoter

createCandidateGenerators

protected List<org.apache.hadoop.hbase.master.balancer.CandidateGenerator> createCandidateGenerators()

Overrides:

createCandidateGenerators in class StochasticLoadBalancer

getRandomGenerator

protected org.apache.hadoop.hbase.master.balancer.CandidateGenerator getRandomGenerator()

Returns any candidate generator in random

Overrides:

getRandomGenerator in class StochasticLoadBalancer

roundRobinAssignment

@NonNull
public Map<ServerName,List<RegionInfo>> roundRobinAssignment(List<RegionInfo> regions,
                                                                      List<ServerName> servers)
                                                               throws HBaseIOException

Round robin assignment: Segregate the regions into two types: 1. The regions that have favored node assignment where at least one of the favored node is still alive. In this case, try to adhere to the current favored nodes assignment as much as possible - i.e., if the current primary is gone, then make the secondary or tertiary as the new host for the region (based on their current load). Note that we don't change the favored node assignments here (even though one or more favored node is currently down). That will be done by the admin operations. 2. The regions that currently don't have favored node assignments. Generate favored nodes for them and then assign. Generate the primary fn in round robin fashion and generate secondary and tertiary as per favored nodes constraints.

Specified by:

roundRobinAssignment in interface LoadBalancer

Overrides:

roundRobinAssignment in class BaseLoadBalancer

Parameters:

regions - all regions

servers - all servers

Returns:

map of server to the regions it should take, or emptyMap if no assignment is possible (ie. no servers)

Throws:

HBaseIOException

randomAssignment

public ServerName randomAssignment(RegionInfo regionInfo,
                                   List<ServerName> servers)
                            throws HBaseIOException

If we have favored nodes for a region, we will return one of the FN as destination. If favored nodes are not present for a region, we will generate and return one of the FN as destination. If we can't generate anything, lets fallback.

Specified by:

randomAssignment in interface LoadBalancer

Overrides:

randomAssignment in class BaseLoadBalancer

Parameters:

regionInfo - Region for which this selection is being done.

Throws:

HBaseIOException

retainAssignment

@NonNull
public Map<ServerName,List<RegionInfo>> retainAssignment(Map<RegionInfo,ServerName> regions,
                                                                  List<ServerName> servers)
                                                           throws HBaseIOException

Reuse BaseLoadBalancer's retainAssignment, but generate favored nodes when its missing.

Specified by:

retainAssignment in interface LoadBalancer

Overrides:

retainAssignment in class BaseLoadBalancer

Parameters:

regions - regions and existing assignment from meta

servers - available servers

Returns:

map of servers and regions to be assigned to them, or emptyMap if no assignment is possible (ie. no servers)

Throws:

HBaseIOException

getFavoredNodes

public List<ServerName> getFavoredNodes(RegionInfo regionInfo)

Specified by:

getFavoredNodes in interface FavoredNodesPromoter

generateFavoredNodesForDaughter

public void generateFavoredNodesForDaughter(List<ServerName> servers,
                                            RegionInfo parent,
                                            RegionInfo regionA,
                                            RegionInfo regionB)
                                     throws IOException

Generate Favored Nodes for daughters during region split.

If the parent does not have FN, regenerates them for the daughters.

If the parent has FN, inherit two FN from parent for each daughter and generate the remaining. The primary FN for both the daughters should be the same as parent. Inherit the secondary FN from the parent but keep it different for each daughter. Choose the remaining FN randomly. This would give us better distribution over a period of time after enough splits.

Specified by:

generateFavoredNodesForDaughter in interface FavoredNodesPromoter

Throws:

IOException

generateFavoredNodesForMergedRegion

public void generateFavoredNodesForMergedRegion(RegionInfo merged,
                                                RegionInfo[] mergeParents)
                                         throws IOException

Generate favored nodes for a region during merge. Choose the FN from one of the sources to keep it simple.

Specified by:

generateFavoredNodesForMergedRegion in interface FavoredNodesPromoter

Throws:

IOException

balanceTable

protected List<RegionPlan> balanceTable(TableName tableName,
                                        Map<ServerName,List<RegionInfo>> loadOfOneTable)

For all regions correctly assigned to favored nodes, we just use the stochastic balancer implementation. For the misplaced regions, we assign a bogus server to it and AM takes care.

Overrides:

balanceTable in class StochasticLoadBalancer

Parameters:

tableName - the table to be balanced

loadOfOneTable - region load of servers for the specific one table

Returns:

List of plans