org.archive.hadoop

Class PerMapOutputFormat<K,V>

java.lang.Object

org.apache.hadoop.mapred.FileOutputFormat<K,V>

org.archive.hadoop.PerMapOutputFormat<K,V>

All Implemented Interfaces:

org.apache.hadoop.mapred.OutputFormat<K,V>

public class PerMapOutputFormat<K,V>
extends org.apache.hadoop.mapred.FileOutputFormat<K,V>

OutputFormat that directs the output to a file named according to the input file. For instance, if the input file is "foo", then the output file is also named "foo". A suffix can be easily added, or a regex+replace applied to the input filename to produce an output filename. This class can be used in conjunction with FilenameInputFormat in a map-reduce job that only does a map() function, no reduce. By combining these input and output formats, it's easy to read from a large set of input files, process each one in a separate map task, and write the output to a file with a name based on the input. For example, suppose you had 1000 WARC files and your map() task just reads a single WARC file and outputs the number of records in it. Use the FilenameInputFormat and the PerMapOutputFormat, setting the "permap.suffix" property to ".count" and for each WARC input file (e.g. "foo.warc.gz") you'll get a corresponding ".count" file in the output (.e.g. "foo.warc.gz.count"). The nice thing about using this class as the OutputFormat is that Hadoop will manage the temporary file for you. This means that if the map task fails (suppose the task node kernel panics), Hadoop will automatically delete the temp file from the failed task and re-schedule it. This class assumes the actual OutputFormat is a SequenceFile. If not -- suppose you want to output a MapFile or plain text -- then specify the output format in the "permap.output.format.class" property. This class was insired by Hadoop's

org.apache.hadoop.mapred.lib.MultipleOutputFormat

Constructor Summary

Constructors

Constructor and Description
PerMapOutputFormat()

Method Summary

Methods

Modifier and Type	Method and Description
void	checkOutputSpecs(org.apache.hadoop.fs.FileSystem ignored, org.apache.hadoop.mapred.JobConf job) Over-ride the default FileOutputFormat's checkOutputSpecs() to allow for the target directory to already exist.
org.apache.hadoop.mapred.RecordWriter<K,V>	getRecordWriter(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.mapred.JobConf job, String name, org.apache.hadoop.util.Progressable progress)

Methods inherited from class org.apache.hadoop.mapred.FileOutputFormat

getCompressOutput, getOutputCompressorClass, getOutputPath, getPathForCustomFile, getTaskOutputPath, getUniqueName, getWorkOutputPath, setCompressOutput, setOutputCompressorClass, setOutputPath

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PerMapOutputFormat

public PerMapOutputFormat()

Method Detail

getRecordWriter

public org.apache.hadoop.mapred.RecordWriter<K,V> getRecordWriter(org.apache.hadoop.fs.FileSystem fs,
                                                         org.apache.hadoop.mapred.JobConf job,
                                                         String name,
                                                         org.apache.hadoop.util.Progressable progress)
                                                           throws IOException

Specified by:

getRecordWriter in interface org.apache.hadoop.mapred.OutputFormat<K,V>

Specified by:

getRecordWriter in class org.apache.hadoop.mapred.FileOutputFormat<K,V>

Throws:

IOException

checkOutputSpecs

public void checkOutputSpecs(org.apache.hadoop.fs.FileSystem ignored,
                    org.apache.hadoop.mapred.JobConf job)
                      throws org.apache.hadoop.mapred.FileAlreadyExistsException,
                             org.apache.hadoop.mapred.InvalidJobConfException,
                             IOException

Over-ride the default FileOutputFormat's checkOutputSpecs() to allow for the target directory to already exist.

Specified by:

checkOutputSpecs in interface org.apache.hadoop.mapred.OutputFormat<K,V>

Overrides:

checkOutputSpecs in class org.apache.hadoop.mapred.FileOutputFormat<K,V>

Throws:

org.apache.hadoop.mapred.FileAlreadyExistsException

org.apache.hadoop.mapred.InvalidJobConfException

IOException