org.archive.url

Class BasicURLCanonicalizer

java.lang.Object

org.archive.url.BasicURLCanonicalizer

All Implemented Interfaces:

URLCanonicalizer

Direct Known Subclasses:

GoogleURLCanonicalizer

public class BasicURLCanonicalizer
extends java.lang.Object
implements URLCanonicalizer

Canonicalizer that does more or less basic fixup. Based initially on rules specified at https://developers.google.com/safe-browsing/developers_guide_v2# Canonicalization. These rules are designed for clients of google's "experimental" Safe Browsing API to "check URLs against Google's constantly-updated blacklists of suspected phishing and malware pages".

This class differs from google in treatment of non-ascii input. Google's rules don't really address this except with one example test case, which seems to suggest taking raw input bytes and pct-encoding them byte for byte. Since the input to this class consists of java strings, not raw bytes, that wouldn't be possible, even if deemed preferable. Instead BasicURLCanonicalizer expresses non-ascii characters pct-encoded UTF-8.

Field Summary

Fields

Modifier and Type	Field and Description
protected static java.nio.charset.Charset	_UTF8

Constructor Summary

Constructors

Constructor and Description
BasicURLCanonicalizer()

Method Summary

Modifier and Type	Method and Description
java.lang.String	attemptIPFormats(java.lang.String host)
void	canonicalize(HandyURL url)
java.lang.String	decode(java.lang.String input)
java.lang.String	escapeOnce(java.lang.String input)
int	getHex(int b)
java.lang.String	minimalEscape(java.lang.String input)
java.lang.String	normalizePath(java.lang.String path)
java.lang.String	unescapeRepeatedly(java.lang.String input)
protected static java.nio.charset.Charset	UTF8()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

_UTF8

protected static java.nio.charset.Charset _UTF8

Constructor Detail

BasicURLCanonicalizer

public BasicURLCanonicalizer()

Method Detail

canonicalize

public void canonicalize(HandyURL url)

Specified by:

canonicalize in interface URLCanonicalizer

normalizePath

public java.lang.String normalizePath(java.lang.String path)

attemptIPFormats

public java.lang.String attemptIPFormats(java.lang.String host)

minimalEscape

public java.lang.String minimalEscape(java.lang.String input)

UTF8

protected static java.nio.charset.Charset UTF8()

escapeOnce

public java.lang.String escapeOnce(java.lang.String input)

unescapeRepeatedly

public java.lang.String unescapeRepeatedly(java.lang.String input)

decode

public java.lang.String decode(java.lang.String input)

getHex

public int getHex(int b)