com.google.cloud.bigtable.mapreduce

Class ShuffledSequenceFileInputFormat<K,V>

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<K,V>

org.apache.hadoop.mapreduce.lib.input.FileInputFormat<K,V>

org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat<K,V>

com.google.cloud.bigtable.mapreduce.ShuffledSequenceFileInputFormat<K,V>

public class ShuffledSequenceFileInputFormat<K,V>
extends org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat<K,V>

An InputFormat randomizes the order of SequenceFile splits. This is needed for the Bigtable import job to spread the load across all of the Bigtable tablet servers.

Version:

$Id: $Id

Author:

igorbernstein2

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

org.apache.hadoop.mapreduce.lib.input.FileInputFormat.Counter

Field Summary

Fields inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

DEFAULT_LIST_STATUS_NUM_THREADS, INPUT_DIR, INPUT_DIR_RECURSIVE, LIST_STATUS_NUM_THREADS, NUM_INPUT_FILES, PATHFILTER_CLASS, SPLIT_MAXSIZE, SPLIT_MINSIZE

Constructor Summary

Constructors

Constructor and Description
ShuffledSequenceFileInputFormat()

Method Summary

Modifier and Type	Method and Description
List<org.apache.hadoop.mapreduce.InputSplit>	getSplits(org.apache.hadoop.mapreduce.JobContext job)

Methods inherited from class org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat

createRecordReader, getFormatMinSplitSize, listStatus

Methods inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

addInputPath, addInputPathRecursively, addInputPaths, computeSplitSize, getBlockIndex, getInputDirRecursive, getInputPathFilter, getInputPaths, getMaxSplitSize, getMinSplitSize, isSplitable, makeSplit, makeSplit, setInputDirRecursive, setInputPathFilter, setInputPaths, setInputPaths, setMaxInputSplitSize, setMinInputSplitSize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ShuffledSequenceFileInputFormat

public ShuffledSequenceFileInputFormat()

Method Detail

getSplits

public List<org.apache.hadoop.mapreduce.InputSplit> getSplits(org.apache.hadoop.mapreduce.JobContext job)
                                                       throws IOException

Overrides:

getSplits in class org.apache.hadoop.mapreduce.lib.input.FileInputFormat<K,V>

Throws:

IOException