Class BasicURLCanonicalizer

java.lang.Object
org.archive.url.BasicURLCanonicalizer
All Implemented Interfaces:
URLCanonicalizer
public class BasicURLCanonicalizer extends Object implements URLCanonicalizer
Canonicalizer that does more or less basic fixup. Based initially on rules specified at https://developers.google.com/safe-browsing/developers_guide_v2# Canonicalization. These rules are designed for clients of Google's "experimental" Safe Browsing API to "check URLs against Google's constantly-updated blacklists of suspected phishing and malware pages".

This class differs from Google in treatment of non-ascii input. Google's rules don't really address this except with one example test case, which seems to suggest taking raw input bytes and pct-encoding them byte for byte. Since the input to this class consists of java strings, not raw bytes, that wouldn't be possible, even if deemed preferable. Instead, BasicURLCanonicalizer expresses non-ascii characters pct-encoded UTF-8.

  • Field Details

    • _UTF8

      protected static Charset _UTF8

  • Constructor Details

    • BasicURLCanonicalizer

      public BasicURLCanonicalizer()

  • Method Details

    • canonicalize

      public void canonicalize(HandyURL url)
      Specified by:
      canonicalize in interface URLCanonicalizer

    • normalizePath

      public String normalizePath(String path)

    • attemptIPFormats

      public String attemptIPFormats(String host)

    • minimalEscape

      public String minimalEscape(String input)

    • UTF8

      protected static Charset UTF8()

    • escapeOnce

      public String escapeOnce(String input)
      Parameters:
      input - String to be percent-encoded. Assumed to be fully unescaped.
      Returns:
      percent-encoded string

    • unescapeRepeatedly

      public String unescapeRepeatedly(String input)

    • decode

      public String decode(String input)

    • getHex

      public int getHex(int b)