org.archive.hadoop

Class FilenameInputFormat

java.lang.Object

org.apache.hadoop.mapred.FileInputFormat<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>

org.archive.hadoop.FilenameInputFormat

All Implemented Interfaces:

org.apache.hadoop.mapred.InputFormat<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>

public class FilenameInputFormat
extends org.apache.hadoop.mapred.FileInputFormat<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>

Handy "input format" which maps the input filename into a "record" which just has the filename. This is very useful for map-reduce jobs where you want to pass the filenames into the map() function. Use this as the input format, and the input filenames will be passed to the map(). The full pathname is given as both the key and the value to the map().

Field Summary

Fields inherited from class org.apache.hadoop.mapred.FileInputFormat

LOG

Constructor Summary

Constructors

Constructor and Description
FilenameInputFormat()

Method Summary

Methods

Modifier and Type	Method and Description
void	configure(org.apache.hadoop.mapred.JobConf conf) Configure per Hadoop properties
org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>	getRecordReader(org.apache.hadoop.mapred.InputSplit genericSplit, org.apache.hadoop.mapred.JobConf job, org.apache.hadoop.mapred.Reporter reporter) Return a RecordReader which returns 1 record: the file path from the InputSplit.
protected boolean	isSplitable(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path file) By definition, not splitable.

Methods inherited from class org.apache.hadoop.mapred.FileInputFormat

addInputPath, addInputPaths, computeSplitSize, getBlockIndex, getInputPathFilter, getInputPaths, getSplitHosts, getSplits, listStatus, setInputPathFilter, setInputPaths, setInputPaths, setMinSplitSize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FilenameInputFormat

public FilenameInputFormat()

Method Detail

configure

public void configure(org.apache.hadoop.mapred.JobConf conf)

Configure per Hadoop properties

isSplitable

protected boolean isSplitable(org.apache.hadoop.fs.FileSystem fs,
                  org.apache.hadoop.fs.Path file)

By definition, not splitable.

Overrides:

isSplitable in class org.apache.hadoop.mapred.FileInputFormat<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>

getRecordReader

public org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text> getRecordReader(org.apache.hadoop.mapred.InputSplit genericSplit,
                                                                                                         org.apache.hadoop.mapred.JobConf job,
                                                                                                         org.apache.hadoop.mapred.Reporter reporter)
                                                                                                           throws IOException

Return a RecordReader which returns 1 record: the file path from the InputSplit.

Specified by:

getRecordReader in interface org.apache.hadoop.mapred.InputFormat<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>

Specified by:

getRecordReader in class org.apache.hadoop.mapred.FileInputFormat<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>

Throws:

IOException