org.archive.url

Class UsableURIFactory

java.lang.Object

org.apache.commons.httpclient.URI

org.archive.url.UsableURIFactory

All Implemented Interfaces:

Serializable, Cloneable, Comparable

public class UsableURIFactory
extends org.apache.commons.httpclient.URI

Factory that returns UsableURIs. Does escaping and fixup on URIs massaging in accordance with RFC2396 and to match browser practice. For example, it removes any '..' if first thing in the path as per IE, converts backslashes preceding the query string to forward slashes, and discards any 'fragment'/anchor portion of the URI. This class will also fail URIs if they are longer than IE's allowed maximum length.

TODO: Test logging.

Author:

stack

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.commons.httpclient.URI

org.apache.commons.httpclient.URI.DefaultCharsetChanged, org.apache.commons.httpclient.URI.LocaleToCharsetMap

Field Summary

Fields

Modifier and Type	Field and Description
protected static String	ACCEPTABLE_ASCII_DOMAIN Characters we'll accept in the domain label part of a URI authority: ASCII letters-digits-hyphen (LDH) plus underscore, with single intervening '.' characters.
static String	APOSTROPH
static String	BACKSLASH
static String	CIRCUMFLEX
static String	CIRCUMFLEX_PATTERN
static char	COLON
static String	COMMERCIAL_AT
static String	DOT
static String	EMPTY_STRING
static String	ESCAPED_APOSTROPH
static String	ESCAPED_BACKSLASH
static String	ESCAPED_CIRCUMFLEX
static String	ESCAPED_LCURBRACKET
static String	ESCAPED_LSQRBRACKET
static String	ESCAPED_PIPE
static String	ESCAPED_QUOT
static String	ESCAPED_RCURBRACKET
static String	ESCAPED_RSQRBRACKET
static String	ESCAPED_SPACE
static String	ESCAPED_SQUOT
static String	HTTP
static String	HTTP_PORT
protected static Pattern	HTTP_SCHEME_SLASHES Pattern that looks for case of three or more slashes after the scheme.
static String	HTTPS
static String	HTTPS_PORT
static String	IMPROPERESC
static String	IMPROPERESC_REPLACE
static String	LCURBRACKET
static String	LCURBRACKET_PATTERN
static String	LSQRBRACKET
static String	LSQRBRACKET_PATTERN
static String	NBSP
static char	PERCENT_SIGN
static String	PIPE
static String	PIPE_PATTERN
protected static Pattern	PORTREGEX Authority port number regex.
static String	QUOT
static String	RCURBRACKET
static String	RCURBRACKET_PATTERN
static Pattern	RFC2396REGEX RFC 2396-inspired regex.
static String	RSQRBRACKET
static String	RSQRBRACKET_PATTERN
static String	SLASH
static String	SLASHDOTDOTSLASH
static String	SPACE
static String	SQUOT
static String	STRAY_SPACING
static String	TRAILING_ESCAPED_SPACE
static String	URI_HEX_ENCODING First percent sign in string followed by two hex chars.

Fields inherited from class org.apache.commons.httpclient.URI

_authority, _fragment, _host, _is_abs_path, _is_hier_part, _is_hostname, _is_IPv4address, _is_IPv6reference, _is_net_path, _is_opaque_part, _is_reg_name, _is_rel_path, _is_server, _opaque, _path, _port, _query, _scheme, _uri, _userinfo, abs_path, absoluteURI, allowed_abs_path, allowed_authority, allowed_fragment, allowed_host, allowed_IPv6reference, allowed_opaque_part, allowed_query, allowed_reg_name, allowed_rel_path, allowed_userinfo, allowed_within_authority, allowed_within_path, allowed_within_query, allowed_within_userinfo, alpha, alphanum, authority, control, defaultDocumentCharset, defaultDocumentCharsetByLocale, defaultDocumentCharsetByPlatform, defaultProtocolCharset, delims, digit, disallowed_opaque_part, disallowed_rel_path, domainlabel, escaped, fragment, hash, hex, hier_part, host, hostname, hostport, IPv4address, IPv6address, IPv6reference, mark, net_path, opaque_part, param, path, path_segments, pchar, percent, port, protocolCharset, query, reg_name, rel_path, rel_segment, relativeURI, reserved, rootPath, scheme, segment, server, space, toplabel, unreserved, unwise, URI_reference, uric, uric_no_slash, userinfo, within_userinfo

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	UsableURIFactory() Protected constructor.

Method Summary

Methods

Modifier and Type	Method and Description
protected void	checkHttpSchemeSpecificPartSlashPrefix(org.apache.commons.httpclient.URI base, String scheme, String schemeSpecificPart) If http(s) scheme, check scheme specific part begins '//'.
protected UsableURI	create(String uri)
protected UsableURI	create(String uri, String charset)
protected UsableURI	create(UsableURI base, String relative)
protected String	escapeWhitespace(String uri) Escape any whitespace found.
static UsableURI	getInstance(String uri)
static UsableURI	getInstance(String uri, String charset)
static UsableURI	getInstance(UsableURI base, String relative)
protected UsableURI	makeOne(String fixedUpUri, boolean escaped, String charset)
protected UsableURI	makeOne(UsableURI base, UsableURI relative)
protected UsableURI	validityCheck(UsableURI uuri) Check the generated UURI.

Methods inherited from class org.apache.commons.httpclient.URI

clone, compareTo, decode, decode, encode, equals, equals, getAboveHierPath, getAuthority, getCurrentHierPath, getDefaultDocumentCharset, getDefaultDocumentCharsetByLocale, getDefaultDocumentCharsetByPlatform, getDefaultProtocolCharset, getEscapedAboveHierPath, getEscapedAuthority, getEscapedCurrentHierPath, getEscapedFragment, getEscapedName, getEscapedPath, getEscapedPathQuery, getEscapedQuery, getEscapedURI, getEscapedURIReference, getEscapedUserinfo, getFragment, getHost, getName, getPath, getPathQuery, getPort, getProtocolCharset, getQuery, getRawAboveHierPath, getRawAuthority, getRawCurrentHierPath, getRawCurrentHierPath, getRawFragment, getRawHost, getRawName, getRawPath, getRawPathQuery, getRawQuery, getRawScheme, getRawURI, getRawURIReference, getRawUserinfo, getScheme, getURI, getURIReference, getUserinfo, hasAuthority, hasFragment, hashCode, hasQuery, hasUserinfo, indexFirstOf, indexFirstOf, indexFirstOf, indexFirstOf, isAbsoluteURI, isAbsPath, isHierPart, isHostname, isIPv4address, isIPv6reference, isNetPath, isOpaquePart, isRegName, isRelativeURI, isRelPath, isServer, normalize, normalize, parseAuthority, parseUriReference, prevalidate, removeFragmentIdentifier, resolvePath, setDefaultDocumentCharset, setDefaultProtocolCharset, setEscapedAuthority, setEscapedFragment, setEscapedPath, setEscapedQuery, setFragment, setPath, setQuery, setRawAuthority, setRawFragment, setRawPath, setRawQuery, setURI, toString, validate, validate

Methods inherited from class java.lang.Object

finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

RFC2396REGEX

public static final Pattern RFC2396REGEX

RFC 2396-inspired regex. From the RFC Appendix B:

 URI Generic Syntax                August 1998

 B. Parsing a URI Reference with a Regular Expression

 As described in Section 4.3, the generic URI syntax is not sufficient
 to disambiguate the components of some forms of URI.  Since the
 "greedy algorithm" described in that section is identical to the
 disambiguation method used by POSIX regular expressions, it is
 natural and commonplace to use a regular expression for parsing the
 potential four components and fragment identifier of a URI reference.

 The following line is the regular expression for breaking-down a URI
 reference into its components.

 ^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?
 12            3  4          5       6  7        8 9

 The numbers in the second line above are only to assist readability;
 they indicate the reference points for each subexpression (i.e., each
 paired parenthesis).  We refer to the value matched for subexpression
  as $.  For example, matching the above expression to

 http://www.ics.uci.edu/pub/ietf/uri/#Related

 results in the following subexpression matches:

 $1 = http:
 $2 = http
 $3 = //www.ics.uci.edu
 $4 = www.ics.uci.edu
 $5 = /pub/ietf/uri/
 $6 = 
 $7 = 
 $8 = #Related
 $9 = Related

 where  indicates that the component is not present, as is
 the case for the query component in the above example.  Therefore, we
 can determine the value of the four components and fragment as

 scheme    = $2
 authority = $4
 path      = $5
 query     = $7
 fragment  = $9
 

--

Below differs from the rfc regex in that... (1) it has java escaping of regex characters (2) we allow a URI made of a fragment only (Added extra group so indexing is off by one after scheme). (3) scheme is limited to legal scheme characters

SLASHDOTDOTSLASH

public static final String SLASHDOTDOTSLASH

See Also:

Constant Field Values

SLASH

public static final String SLASH

See Also:

Constant Field Values

HTTP

public static final String HTTP

See Also:

Constant Field Values

HTTP_PORT

public static final String HTTP_PORT

See Also:

Constant Field Values

HTTPS

public static final String HTTPS

See Also:

Constant Field Values

HTTPS_PORT

public static final String HTTPS_PORT

See Also:

Constant Field Values

DOT

public static final String DOT

See Also:

Constant Field Values

EMPTY_STRING

public static final String EMPTY_STRING

See Also:

Constant Field Values

NBSP

public static final String NBSP

See Also:

Constant Field Values

SPACE

public static final String SPACE

See Also:

Constant Field Values

ESCAPED_SPACE

public static final String ESCAPED_SPACE

See Also:

Constant Field Values

TRAILING_ESCAPED_SPACE

public static final String TRAILING_ESCAPED_SPACE

See Also:

Constant Field Values

PIPE

public static final String PIPE

See Also:

Constant Field Values

PIPE_PATTERN

public static final String PIPE_PATTERN

See Also:

Constant Field Values

ESCAPED_PIPE

public static final String ESCAPED_PIPE

See Also:

Constant Field Values

CIRCUMFLEX

public static final String CIRCUMFLEX

See Also:

Constant Field Values

CIRCUMFLEX_PATTERN

public static final String CIRCUMFLEX_PATTERN

See Also:

Constant Field Values

ESCAPED_CIRCUMFLEX

public static final String ESCAPED_CIRCUMFLEX

See Also:

Constant Field Values

QUOT

public static final String QUOT

See Also:

Constant Field Values

ESCAPED_QUOT

public static final String ESCAPED_QUOT

See Also:

Constant Field Values

SQUOT

public static final String SQUOT

See Also:

Constant Field Values

ESCAPED_SQUOT

public static final String ESCAPED_SQUOT

See Also:

Constant Field Values

APOSTROPH

public static final String APOSTROPH

See Also:

Constant Field Values

ESCAPED_APOSTROPH

public static final String ESCAPED_APOSTROPH

See Also:

Constant Field Values

LSQRBRACKET

public static final String LSQRBRACKET

See Also:

Constant Field Values

LSQRBRACKET_PATTERN

public static final String LSQRBRACKET_PATTERN

See Also:

Constant Field Values

ESCAPED_LSQRBRACKET

public static final String ESCAPED_LSQRBRACKET

See Also:

Constant Field Values

RSQRBRACKET

public static final String RSQRBRACKET

See Also:

Constant Field Values

RSQRBRACKET_PATTERN

public static final String RSQRBRACKET_PATTERN

See Also:

Constant Field Values

ESCAPED_RSQRBRACKET

public static final String ESCAPED_RSQRBRACKET

See Also:

Constant Field Values

LCURBRACKET

public static final String LCURBRACKET

See Also:

Constant Field Values

LCURBRACKET_PATTERN

public static final String LCURBRACKET_PATTERN

See Also:

Constant Field Values

ESCAPED_LCURBRACKET

public static final String ESCAPED_LCURBRACKET

See Also:

Constant Field Values

RCURBRACKET

public static final String RCURBRACKET

See Also:

Constant Field Values

RCURBRACKET_PATTERN

public static final String RCURBRACKET_PATTERN

See Also:

Constant Field Values

ESCAPED_RCURBRACKET

public static final String ESCAPED_RCURBRACKET

See Also:

Constant Field Values

BACKSLASH

public static final String BACKSLASH

See Also:

Constant Field Values

ESCAPED_BACKSLASH

public static final String ESCAPED_BACKSLASH

See Also:

Constant Field Values

STRAY_SPACING

public static final String STRAY_SPACING

See Also:

Constant Field Values

IMPROPERESC_REPLACE

public static final String IMPROPERESC_REPLACE

See Also:

Constant Field Values

IMPROPERESC

public static final String IMPROPERESC

See Also:

Constant Field Values

COMMERCIAL_AT

public static final String COMMERCIAL_AT

See Also:

Constant Field Values

PERCENT_SIGN

public static final char PERCENT_SIGN

See Also:

Constant Field Values

COLON

public static final char COLON

See Also:

Constant Field Values

URI_HEX_ENCODING

public static final String URI_HEX_ENCODING

First percent sign in string followed by two hex chars.

See Also:

Constant Field Values

PORTREGEX

protected static final Pattern PORTREGEX

Authority port number regex.

ACCEPTABLE_ASCII_DOMAIN

protected static final String ACCEPTABLE_ASCII_DOMAIN

Characters we'll accept in the domain label part of a URI authority: ASCII letters-digits-hyphen (LDH) plus underscore, with single intervening '.' characters. (We accept '_' because DNS servers have tolerated for many years counter to spec; we also accept dash patterns and ACE prefixes that will be rejected by IDN-punycoding attempt.)

See Also:

Constant Field Values

HTTP_SCHEME_SLASHES

protected static final Pattern HTTP_SCHEME_SLASHES

Pattern that looks for case of three or more slashes after the scheme. If found, we replace them with two only as mozilla does.

Constructor Detail

UsableURIFactory

protected UsableURIFactory()

Protected constructor.

Method Detail

getInstance

public static UsableURI getInstance(String uri)
                             throws org.apache.commons.httpclient.URIException

Parameters:

uri - URI as string.

Returns:

An instance of UURI

Throws:

org.apache.commons.httpclient.URIException

getInstance

public static UsableURI getInstance(String uri,
                    String charset)
                             throws org.apache.commons.httpclient.URIException

Parameters:

uri - URI as string.

charset - Character encoding of the passed uri string.

Returns:

An instance of UURI

Throws:

org.apache.commons.httpclient.URIException

getInstance

public static UsableURI getInstance(UsableURI base,
                    String relative)
                             throws org.apache.commons.httpclient.URIException

Parameters:

base - Base uri to use resolving passed relative uri.

relative - URI as string.

Returns:

An instance of UURI

Throws:

org.apache.commons.httpclient.URIException

create

protected UsableURI create(String uri)
                    throws org.apache.commons.httpclient.URIException

Parameters:

uri - URI as string.

Returns:

Instance of UURI.

Throws:

org.apache.commons.httpclient.URIException

create

protected UsableURI create(String uri,
               String charset)
                    throws org.apache.commons.httpclient.URIException

Parameters:

uri - URI as string.

charset - Original encoding of the string.

Returns:

Instance of UURI.

Throws:

org.apache.commons.httpclient.URIException

makeOne

protected UsableURI makeOne(String fixedUpUri,
                boolean escaped,
                String charset)
                     throws org.apache.commons.httpclient.URIException

Throws:

org.apache.commons.httpclient.URIException

makeOne

protected UsableURI makeOne(UsableURI base,
                UsableURI relative)
                     throws org.apache.commons.httpclient.URIException

Throws:

org.apache.commons.httpclient.URIException

create

protected UsableURI create(UsableURI base,
               String relative)
                    throws org.apache.commons.httpclient.URIException

Parameters:

base - UURI to use as a base resolving relative.

relative - Relative URI.

Returns:

Instance of UURI.

Throws:

org.apache.commons.httpclient.URIException

validityCheck

protected UsableURI validityCheck(UsableURI uuri)
                           throws org.apache.commons.httpclient.URIException

Check the generated UURI. At the least look at length of uuri string. We were seeing case where before escaping, string was < MAX_URL_LENGTH but after was >. Letting out a too-big message was causing us troubles later down the processing chain.

Parameters:

uuri - Created uuri to check.

Returns:

The passed uuri so can easily inline this check.

Throws:

org.apache.commons.httpclient.URIException

checkHttpSchemeSpecificPartSlashPrefix

protected void checkHttpSchemeSpecificPartSlashPrefix(org.apache.commons.httpclient.URI base,
                                          String scheme,
                                          String schemeSpecificPart)
                                               throws org.apache.commons.httpclient.URIException

If http(s) scheme, check scheme specific part begins '//'.

Throws:

org.apache.commons.httpclient.URIException

See Also:

Section 3.1. Common Internet Scheme Syntax

escapeWhitespace

protected String escapeWhitespace(String uri)

Escape any whitespace found. The parent class takes care of the bulk of escaping. But if any instance of escaping is found in the URI, then we ask for parent to do NO escaping. Here we escape any whitespace found irrespective of whether the uri has already been escaped. We do this for case where uri has been judged already-escaped only, its been incompletly done and whitespace remains. Spaces, etc., in the URI are a real pain. Their presence will break log file and ARC parsing.

Parameters:

uri - URI string to check.

Returns:

uri with spaces escaped if any found.