edu.umd.cloud9.webgraph.driver
Class ClueWebDriver

java.lang.Object
  org.apache.hadoop.conf.Configured
      edu.umd.cloud9.webgraph.driver.ClueWebDriver

All Implemented Interfaces:

Configurable, Tool

public class ClueWebDriver
extends Configured
implements Tool

Main driver program for extracting the ClueWeb09 web graph, reverse web graph, and lines of anchor text. Command-line arguments are as follows:

• [collection-base-path]: the base path to the ClueWeb09 collection
• [output-base-path]: the base path under which the output would be stored
• [docno-mapping-file]: the path to the docno-mapping.dat file
• [from-segment]: starting segment number
• [to-segment]: ending segment number
• [includeInternalLinks?]: 1 for including the internal links (i.e., links within a domain), 0 for not
• [compute-default-anchor-weight?]: 1 to compute the default weights for lines of external anchor text, 0 for not
• [normalizer] A normalizer class used to normalize the lines of anchor text, must extend *.anchor.normalize.AnchorTextNormalizer.

The starting and ending segments will correspond to paths /[collection-base-path]/en.XX to /[collection-base-path]/en.YY.

The default weight used in this program was originally proposed by Metzler et. al in the following paper:
D. Metzler, J. Novak, H. Cui, and S. Reddy. Building enriched document representations using aggregated anchor text. In Proc. 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 219{226, New York, NY, USA, 2009. ACM.

Author:

Nima Asadi

Field Summary

static String	outputExtractLinks
static String	outputHostnames
static String	outputReverseWebGraph
static String	outputWebGraph
static String	outputWeightedReverseWebGraph

Constructor Summary

ClueWebDriver()

Method Summary

static void	main(String[] args)
int	run(String[] args)

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Detail

outputExtractLinks

public static final String outputExtractLinks

See Also:

Constant Field Values

outputReverseWebGraph

public static final String outputReverseWebGraph

See Also:

Constant Field Values

outputWebGraph

public static final String outputWebGraph

See Also:

Constant Field Values

outputHostnames

public static final String outputHostnames

See Also:

Constant Field Values

outputWeightedReverseWebGraph

public static final String outputWeightedReverseWebGraph

See Also:

Constant Field Values

Constructor Detail

ClueWebDriver

public ClueWebDriver()

Method Detail

run

public int run(String[] args)
        throws Exception

Specified by:

run in interface Tool

Throws:

Exception

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception