org.archive.url

Class BasicURLCanonicalizer

java.lang.Object

org.archive.url.BasicURLCanonicalizer

All Implemented Interfaces:

URLCanonicalizer

Direct Known Subclasses:

GoogleURLCanonicalizer

public class BasicURLCanonicalizer
extends Object
implements URLCanonicalizer

Canonicalizer that does more or less basic fixup. Based initially on rules specified at https://developers.google.com/safe-browsing/developers_guide_v2# Canonicalization. These rules are designed for clients of google's "experimental" Safe Browsing API to "check URLs against Google's constantly-updated blacklists of suspected phishing and malware pages".

This class differs from google in treatment of non-ascii input. Google's rules don't really address this except with one example test case, which seems to suggest taking raw input bytes and pct-encoding them byte for byte. Since the input to this class consists of java strings, not raw bytes, that wouldn't be possible, even if deemed preferable. Instead BasicURLCanonicalizer expresses non-ascii characters pct-encoded UTF-8.

Field Summary

Fields

Modifier and Type	Field and Description
protected static Charset	_UTF8

Constructor Summary

Constructors

Constructor and Description
BasicURLCanonicalizer()

Method Summary

Methods

Modifier and Type	Method and Description
String	attemptIPFormats(String host)
void	canonicalize(HandyURL url)
String	decode(String input)
String	escapeOnce(String input)
int	getHex(int b)
String	minimalEscape(String input)
String	normalizePath(String path)
String	unescapeRepeatedly(String input)
protected static Charset	UTF8()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

_UTF8

protected static Charset _UTF8

Constructor Detail

BasicURLCanonicalizer

public BasicURLCanonicalizer()

Method Detail

canonicalize

public void canonicalize(HandyURL url)

Specified by:

canonicalize in interface URLCanonicalizer

normalizePath

public String normalizePath(String path)

attemptIPFormats

public String attemptIPFormats(String host)

minimalEscape

public String minimalEscape(String input)

UTF8

protected static Charset UTF8()

escapeOnce

public String escapeOnce(String input)

unescapeRepeatedly

public String unescapeRepeatedly(String input)

decode

public String decode(String input)

getHex

public int getHex(int b)