org.archive.util

Class SurtPrefixSet

java.lang.Object

java.util.AbstractCollection<E>

java.util.AbstractSet<E>

java.util.concurrent.ConcurrentSkipListSet<java.lang.String>

org.archive.util.PrefixSet

org.archive.util.SurtPrefixSet

All Implemented Interfaces:

java.io.Serializable, java.lang.Cloneable, java.lang.Iterable<java.lang.String>, java.util.Collection<java.lang.String>, java.util.NavigableSet<java.lang.String>, java.util.Set<java.lang.String>, java.util.SortedSet<java.lang.String>

public class SurtPrefixSet
extends PrefixSet

Specialized TreeSet for keeping a set of String prefixes. Redundant prefixes (those that are themselves prefixed by other set entries) are eliminated.

Author:

gojomo

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
SurtPrefixSet()

Method Summary

Modifier and Type	Method and Description
void	addFromPlain(java.lang.String u) Given a plain URI or hostname, deduce an implied SURT prefix from it and add to active prefixes.
static java.lang.String	asPrefix(java.lang.String s) Utility method for truncating a SURT that came from a full URI (as a seed, for example) into a prefix for determining inclusion.
boolean	considerAsAddDirective(java.lang.String suri) Interpret the given SURT/URI/host as a directive, returning true if it was meaningful.
void	convertAllPrefixesToDomains() Changes all prefixes so that they only enforce a general domain (allowing subdomains).For prefixes that don't include a ')', no change is necessary.
void	convertAllPrefixesToHosts() Changes all prefixes so that they enforce an exact host.
static java.lang.String	convertPrefixToDomain(java.lang.String prefix)
static java.lang.String	convertPrefixToHost(java.lang.String prefix)
void	exportTo(java.io.FileWriter fw)
static java.lang.String	getCandidateSurt(UsableURI u) Calculate the SURT form URI to use as a candidate against prefixes from the given Object (CandidateURI or UURI)
void	importFrom(java.io.Reader r) Read a set of SURT prefixes from a reader source; keep sorted and with redundant entries removed.
void	importFromMixed(java.io.Reader r, boolean deduceFromSeeds) Import SURT prefixes from a reader with mixed URI and SURT prefix format.
void	importFromUris(java.io.Reader r)
static void	main(java.lang.String[] args) Allow class to be used as a command-line tool for converting URL lists (or naked host or host/path fragments implied to be HTTP URLs) to implied SURT prefix form.
static java.lang.String	prefixFromPlainForceHttp(java.lang.String u) Given a plain URI or hostname/hostname+path, deduce an implied SURT prefix from it.

Methods inherited from class org.archive.util.PrefixSet

add, containsPrefixOf

Methods inherited from class java.util.concurrent.ConcurrentSkipListSet

ceiling, clear, clone, comparator, contains, descendingIterator, descendingSet, equals, first, floor, headSet, headSet, higher, isEmpty, iterator, last, lower, pollFirst, pollLast, remove, removeAll, size, spliterator, subSet, subSet, tailSet, tailSet

Methods inherited from class java.util.AbstractSet

hashCode

Methods inherited from class java.util.AbstractCollection

addAll, containsAll, retainAll, toArray, toArray, toString

Methods inherited from class java.lang.Object

finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.util.Set

addAll, containsAll, hashCode, retainAll, toArray, toArray

Methods inherited from interface java.util.Collection

parallelStream, removeIf, stream

Methods inherited from interface java.lang.Iterable

forEach

Constructor Detail

SurtPrefixSet

public SurtPrefixSet()

Method Detail

importFrom

public void importFrom(java.io.Reader r)

Read a set of SURT prefixes from a reader source; keep sorted and with redundant entries removed.

Parameters:

r - reader over file of SURT_format strings

Throws:

java.io.IOException

importFromUris

public void importFromUris(java.io.Reader r)

Parameters:

r - Where to read from.

importFromMixed

public void importFromMixed(java.io.Reader r,
                            boolean deduceFromSeeds)

Import SURT prefixes from a reader with mixed URI and SURT prefix format.

Parameters:

r - the reader to import the prefixes from

deduceFromSeeds - true to also import SURT prefixes implied from normal URIs/hostname seeds

considerAsAddDirective

public boolean considerAsAddDirective(java.lang.String suri)

Interpret the given SURT/URI/host as a directive, returning true if it was meaningful.

Parameters:

suri - potential directive

Returns:

boolean true if applied as directive, false otherwise

addFromPlain

public void addFromPlain(java.lang.String u)

Given a plain URI or hostname, deduce an implied SURT prefix from it and add to active prefixes.

Parameters:

u - String of URI or hostname

prefixFromPlainForceHttp

public static java.lang.String prefixFromPlainForceHttp(java.lang.String u)

Given a plain URI or hostname/hostname+path, deduce an implied SURT prefix from it. Results may be unpredictable on strings that cannot be interpreted as URIs. UURI 'fixup' is applied to the URI that is built. HTTPS URIs are changed to HTTP as a convenience for the usual preferred common-treatment.

Parameters:

u - URI or almost-URI to consider

Returns:

implied SURT prefix form

asPrefix

public static java.lang.String asPrefix(java.lang.String s)

Utility method for truncating a SURT that came from a full URI (as a seed, for example) into a prefix for determining inclusion. This involves:

    (1) removing the last path component, if any
        (anything after the last '/', if there are
        at least 3 '/'s)
    (2) removing a trailing ')', if present, opening
        the possibility of proper subdomains. (This
        means that the presence or absence of a
        trailing '/' after a hostname in a seed list
        is significant for the how the SURT prefix is 
        created, even though it is not signficant for 
        the URI's treatment as a seed.)
 

Parameters:

s - String to work on.

Returns:

As prefix.

getCandidateSurt

public static java.lang.String getCandidateSurt(UsableURI u)

Calculate the SURT form URI to use as a candidate against prefixes from the given Object (CandidateURI or UURI)

Parameters:

object - CandidateURI or UURI

Returns:

SURT form of URI for evaluation, or null if unavailable

exportTo

public void exportTo(java.io.FileWriter fw)
              throws java.io.IOException

Parameters:

fw -

Throws:

java.io.IOException

convertAllPrefixesToHosts

public void convertAllPrefixesToHosts()

Changes all prefixes so that they enforce an exact host. For prefixes that already include a ')', this means discarding anything after ')' (path info). For prefixes that don't include a ')' -- domain prefixes open to subdomains -- add the closing ')' (or ",)").

convertPrefixToHost

public static java.lang.String convertPrefixToHost(java.lang.String prefix)

convertAllPrefixesToDomains

public void convertAllPrefixesToDomains()

Changes all prefixes so that they only enforce a general domain (allowing subdomains).For prefixes that don't include a ')', no change is necessary. For others, truncate everything from the ')' onward. Additionally, truncate off "www," if it appears.

convertPrefixToDomain

public static java.lang.String convertPrefixToDomain(java.lang.String prefix)

main

public static void main(java.lang.String[] args)
                 throws java.io.IOException

Allow class to be used as a command-line tool for converting URL lists (or naked host or host/path fragments implied to be HTTP URLs) to implied SURT prefix form. Read from stdin or first file argument. Writes to stdout.

Parameters:

args - cmd-line arguments: may include input file

Throws:

java.io.IOException