Class SurtPrefixSet

java.lang.Object
java.util.AbstractCollection<String>
java.util.AbstractSet<String>
java.util.concurrent.ConcurrentSkipListSet<String>
org.archive.util.PrefixSet
org.archive.util.SurtPrefixSet
All Implemented Interfaces:
Serializable, Cloneable, Iterable<String>, Collection<String>, NavigableSet<String>, Set<String>, SortedSet<String>
public class SurtPrefixSet extends PrefixSet
Specialized TreeSet for keeping a set of String prefixes. Redundant prefixes (those that are themselves prefixed by other set entries) are eliminated.
Author:
gojomo
See Also:

  • Constructor Details

    • SurtPrefixSet

      public SurtPrefixSet()

  • Method Details

    • importFrom

      public void importFrom(Reader r)
      Read a set of SURT prefixes from a reader source; keep sorted and with redundant entries removed.
      Parameters:
      r - reader over file of SURT_format strings

    • importFromUris

      public void importFromUris(Reader r)
      Parameters:
      r - Where to read from.

    • importFromMixed

      public void importFromMixed(Reader r, boolean deduceFromSeeds)
      Import SURT prefixes from a reader with mixed URI and SURT prefix format.
      Parameters:
      r - the reader to import the prefixes from
      deduceFromSeeds - true to also import SURT prefixes implied from normal URIs/hostname seeds

    • considerAsAddDirective

      public boolean considerAsAddDirective(String suri)
      Interpret the given SURT/URI/host as a directive, returning true if it was meaningful.
      Parameters:
      suri - potential directive
      Returns:
      boolean true if applied as directive, false otherwise

    • addFromPlain

      public void addFromPlain(String u)
      Given a plain URI or hostname, deduce an implied SURT prefix from it and add to active prefixes.
      Parameters:
      u - String of URI or hostname

    • prefixFromPlainForceHttp

      public static String prefixFromPlainForceHttp(String u)
      Given a plain URI or hostname/hostname+path, deduce an implied SURT prefix from it. Results may be unpredictable on strings that cannot be interpreted as URIs. UURI 'fixup' is applied to the URI that is built. HTTPS URIs are changed to HTTP as a convenience for the usual preferred common-treatment.
      Parameters:
      u - URI or almost-URI to consider
      Returns:
      implied SURT prefix form

    • asPrefix

      public static String asPrefix(String s)
      Utility method for truncating a SURT that came from a full URI (as a seed, for example) into a prefix for determining inclusion. This involves: 
          (1) removing the last path component, if any
        (anything after the last '/', if there are
        at least 3 '/'s)
    (2) removing a trailing ')', if present, opening
        the possibility of proper subdomains. (This
        means that the presence or absence of a
        trailing '/' after a hostname in a seed list
        is significant for the how the SURT prefix is 
        created, even though it is not signficant for 
        the URI's treatment as a seed.)
      Parameters:
      s - String to work on.
      Returns:
      As prefix.

    • getCandidateSurt

      public static String getCandidateSurt(UsableURI u)
      Calculate the SURT form URI to use as a candidate against prefixes from the given Object (CandidateURI or UURI)
      Parameters:
      u - CandidateURI or UURI
      Returns:
      SURT form of URI for evaluation, or null if unavailable

    • exportTo

      public void exportTo(FileWriter fw) throws IOException
      Parameters:
      fw -
      Throws:
      IOException

    • convertAllPrefixesToHosts

      public void convertAllPrefixesToHosts()
      Changes all prefixes so that they enforce an exact host. For prefixes that already include a ')', this means discarding anything after ')' (path info). For prefixes that don't include a ')' -- domain prefixes open to subdomains -- add the closing ')' (or ",)").

    • convertPrefixToHost

      public static String convertPrefixToHost(String prefix)

    • convertAllPrefixesToDomains

      public void convertAllPrefixesToDomains()
      Changes all prefixes so that they only enforce a general domain (allowing subdomains).For prefixes that don't include a ')', no change is necessary. For others, truncate everything from the ')' onward. Additionally, truncate off "www," if it appears.

    • convertPrefixToDomain

      public static String convertPrefixToDomain(String prefix)

    • main

      public static void main(String[] args) throws IOException
      Allow class to be used as a command-line tool for converting URL lists (or naked host or host/path fragments implied to be HTTP URLs) to implied SURT prefix form. Read from stdin or first file argument. Writes to stdout.
      Parameters:
      args - cmd-line arguments: may include input file
      Throws:
      IOException