org.archive.util

Class SURT

java.lang.Object

org.archive.util.SURT

public class SURT
extends Object

Sort-friendly URI Reordering Transform. Converts URIs of the form: scheme://userinfo@domain.tld:port/path?query#fragment ...into... scheme://(tld,domain,:port@userinfo)/path?query#fragment The '(' ')' characters serve as an unambiguous notice that the so-called 'authority' portion of the URI ([userinfo@]host[:port] in http URIs) has been transformed; the commas prevent confusion with regular hostnames. This remedies the 'problem' with standard URIs that the host portion of a regular URI, with its dotted-domains, is actually in reverse order from the natural hierarchy that's usually helpful for grouping and sorting. The value of respecting URI case variance is considered negligible: it is vanishingly rare for case-variance to be meaningful, while URI case- variance often arises from people's confusion or sloppiness, and they only correct it insofar as necessary to avoid blatant problems. Thus the usual SURT form is considered to be flattened to all lowercase, and not completely reversible.

Author:

gojomo

Field Summary

Fields

Modifier and Type	Field and Description
protected static String	BEGIN_TRANSFORMED_AUTHORITY
protected static char	DOT
protected static String	END_TRANSFORMED_AUTHORITY
protected static String	TRANSFORMED_HOST_DELIM
protected static String	URI_SPLITTER

Constructor Summary

Constructors

Constructor and Description
SURT()

Method Summary

Methods

Modifier and Type	Method and Description
static String	fromPlain(String u) Given a plain URI or hostname/hostname+path, give its SURT form.
static String	fromURI(String s) Utility method for creating the SURT form of the URI in the given String.
static String	fromURI(String s, boolean preserveCase) Utility method for creating the SURT form of the URI in the given String.
static void	main(String[] args) Allow class to be used as a command-line tool for converting URL lists (or naked host or host/path fragments implied to be HTTP URLs) to SURT form.
static String	prefixFromPlain(String u) Given a plain URI or hostname/hostname+path, deduce an implied SURT prefix from it.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DOT

protected static char DOT

BEGIN_TRANSFORMED_AUTHORITY

protected static String BEGIN_TRANSFORMED_AUTHORITY

TRANSFORMED_HOST_DELIM

protected static String TRANSFORMED_HOST_DELIM

END_TRANSFORMED_AUTHORITY

protected static String END_TRANSFORMED_AUTHORITY

URI_SPLITTER

protected static String URI_SPLITTER

Constructor Detail

SURT

public SURT()

Method Detail

fromURI

public static String fromURI(String s)

Utility method for creating the SURT form of the URI in the given String. By default, does not preserve casing.

Parameters:

s - String URI to be converted to SURT form

Returns:

SURT form

fromURI

public static String fromURI(String s,
             boolean preserveCase)

Utility method for creating the SURT form of the URI in the given String. If it appears a bit convoluted in its approach, note that it was optimized to minimize object-creation after allocation-sites profiling indicated this method was a top source of garbage in long-running crawls. Assumes that the String URI has already been cleaned/fixed (eg by UURI fixup) in ways that put it in its crawlable form for evaluation.

Parameters:

s - String URI to be converted to SURT form

preserveCase - whether original case should be preserved

Returns:

SURT form

prefixFromPlain

public static String prefixFromPlain(String u)

Given a plain URI or hostname/hostname+path, deduce an implied SURT prefix from it. Results may be unpredictable on strings that cannot be interpreted as URIs. UURI 'fixup' is applied to the URI that is built.

Parameters:

u - URI or almost-URI to consider

Returns:

implied SURT prefix form

fromPlain

public static String fromPlain(String u)

Given a plain URI or hostname/hostname+path, give its SURT form. Results may be unpredictable on strings that cannot be interpreted as URIs. UURI 'fixup' is applied to the URI before conversion to SURT form.

Parameters:

u - URI or almost-URI to consider

Returns:

implied SURT prefix form

main

public static void main(String[] args)
                 throws IOException

Allow class to be used as a command-line tool for converting URL lists (or naked host or host/path fragments implied to be HTTP URLs) to SURT form. Lines that cannot be converted are returned unchanged. Read from stdin or first file argument. Writes to stdout or second argument filename

Parameters:

args - cmd-line arguments

Throws:

IOException