Class ArchiveUtils

java.lang.Object
org.archive.util.ArchiveUtils
public class ArchiveUtils extends Object
Miscellaneous useful methods.
Author:
gojomo & others

  • Field Details

    • VERSION

      public static final String VERSION

    • MAX_INT_CHAR_WIDTH

      public static int MAX_INT_CHAR_WIDTH

    • LAST_UNIQUE_NOW17

      protected static long LAST_UNIQUE_NOW17

    • LAST_TIMESTAMP17

      protected static String LAST_TIMESTAMP17

    • LAST_UNIQUE_NOW14

      protected static long LAST_UNIQUE_NOW14

    • LAST_TIMESTAMP14

      protected static String LAST_TIMESTAMP14

    • TLDS

      public static Set<String> TLDS

    • GZIP_SUFFIX

      public static final String GZIP_SUFFIX
      suffix to recognize gzipped files
      See Also:

  • Constructor Details

    • ArchiveUtils

      public ArchiveUtils()

  • Method Details

    • get17DigitDate

      public static String get17DigitDate()
      Utility function for creating arc-style date stamps in the format yyyMMddHHmmssSSS. Date stamps are in the UTC time zone
      Returns:
      the date stamp

    • getUnique17DigitDate

      public static String getUnique17DigitDate()
      Utility function for creating UNIQUE-from-this-class arc-style date stamps in the format yyyMMddHHmmssSSS. Rather than giving a duplicate datestamp on a subsequent call, will increment the milliseconds until a unique value is returned. Date stamps are in the UTC time zone
      Returns:
      the date stamp

    • get14DigitDate

      public static String get14DigitDate()
      Utility function for creating arc-style date stamps in the format yyyyMMddHHmmss. Date stamps are in the UTC time zone
      Returns:
      the date stamp

    • getUnique14DigitDate

      public static String getUnique14DigitDate()
      Utility function for creating UNIQUE-from-this-class arc-style date stamps in the format yyyMMddHHmmss. Rather than giving a duplicate datestamp on a subsequent call, will increment the seconds until a unique value is returned. Date stamps are in the UTC time zone
      Returns:
      the date stamp

    • get12DigitDate

      public static String get12DigitDate()
      Utility function for creating arc-style date stamps in the format yyyyMMddHHmm. Date stamps are in the UTC time zone
      Returns:
      the date stamp

    • getLog17Date

      public static String getLog17Date()
      Utility function for creating log timestamps, in W3C/ISO8601 format, assuming UTC. Use current time. Format is yyyy-MM-dd'T'HH:mm:ss.SSS'Z'
      Returns:
      the date stamp

    • getLog17Date

      public static String getLog17Date(long date)
      Utility function for creating log timestamps, in W3C/ISO8601 format, assuming UTC. Format is yyyy-MM-dd'T'HH:mm:ss.SSS'Z'
      Parameters:
      date - Date to format.
      Returns:
      the date stamp

    • getLog14Date

      public static String getLog14Date()
      Utility function for creating log timestamps, in W3C/ISO8601 format, assuming UTC. Use current time. Format is yyyy-MM-dd'T'HH:mm:ss'Z'
      Returns:
      the date stamp

    • getLog14Date

      public static String getLog14Date(long date)
      Utility function for creating log timestamps, in W3C/ISO8601 format, assuming UTC. Format is yyyy-MM-dd'T'HH:mm:ss'Z'
      Parameters:
      date - long timestamp to format.
      Returns:
      the date stamp

    • getLog14Date

      public static String getLog14Date(Date date)
      Utility function for creating log timestamps, in W3C/ISO8601 format, assuming UTC. Format is yyyy-MM-dd'T'HH:mm:ss'Z'
      Parameters:
      date - Date to format.
      Returns:
      the date stamp

    • parse14DigitISODate

      public static Date parse14DigitISODate(String datetime, Date defaultVal)

    • get17DigitDate

      public static String get17DigitDate(long date)
      Utility function for creating arc-style date stamps in the format yyyyMMddHHmmssSSS. Date stamps are in the UTC time zone
      Parameters:
      date - milliseconds since epoc
      Returns:
      the date stamp

    • get17DigitDate

      public static String get17DigitDate(Date date)

    • get14DigitDate

      public static String get14DigitDate(long date)
      Utility function for creating arc-style date stamps in the format yyyyMMddHHmmss. Date stamps are in the UTC time zone
      Parameters:
      date - milliseconds since epoc
      Returns:
      the date stamp

    • get14DigitDate

      public static String get14DigitDate(Date d)

    • get12DigitDate

      public static String get12DigitDate(long date)
      Utility function for creating arc-style date stamps in the format yyyyMMddHHmm. Date stamps are in the UTC time zone
      Parameters:
      date - milliseconds since epoc
      Returns:
      the date stamp

    • get12DigitDate

      public static String get12DigitDate(Date d)

    • getDate

      public static Date getDate(String d, Date defaultDate)
      A version of getDate which returns the default instead of throwing an exception if parsing fails

    • getDate

      public static Date getDate(String d) throws ParseException
      Parses an ARC-style date. If passed String is < 12 characters in length, we pad. At a minimum, String should contain a year (>=4 characters). Parse will also fail if day or month are incompletely specified. Depends on the above getXXDigitDate methods.
      Parameters:
      d - A 4-17 digit date in ARC style (yyyy to yyyyMMddHHmmssSSS) formatting.
      Returns:
      A Date object representing the passed String.
      Throws:
      ParseException

    • dateToTimestamp

      public static String dateToTimestamp(String input)
      Convert a user-entered date into a timestamp
      Parameters:
      input -
      Returns:

    • parse17DigitDate

      public static Date parse17DigitDate(String date) throws ParseException
      Utility function for parsing arc-style date stamps in the format yyyMMddHHmmssSSS. Date stamps are in the UTC time zone. The whole string will not be parsed, only the first 17 digits.
      Parameters:
      date - an arc-style formatted date stamp
      Returns:
      the Date corresponding to the date stamp string
      Throws:
      ParseException - if the inputstring was malformed

    • parse14DigitDate

      public static Date parse14DigitDate(String date) throws ParseException
      Utility function for parsing arc-style date stamps in the format yyyMMddHHmmss. Date stamps are in the UTC time zone. The whole string will not be parsed, only the first 14 digits.
      Parameters:
      date - an arc-style formatted date stamp
      Returns:
      the Date corresponding to the date stamp string
      Throws:
      ParseException - if the inputstring was malformed

    • parse12DigitDate

      public static Date parse12DigitDate(String date) throws ParseException
      Utility function for parsing arc-style date stamps in the format yyyMMddHHmm. Date stamps are in the UTC time zone. The whole string will not be parsed, only the first 12 digits.
      Parameters:
      date - an arc-style formatted date stamp
      Returns:
      the Date corresponding to the date stamp string
      Throws:
      ParseException - if the inputstring was malformed

    • secondsSinceEpoch

      public static String secondsSinceEpoch(String timestamp) throws ParseException
      Parameters:
      timestamp - A 14-digit timestamp or the suffix for a 14-digit timestamp: E.g. '20010909014640' or '20010101' or '1970'.
      Returns:
      Seconds since the epoch as a string zero-pre-padded so always Integer.MAX_VALUE wide (Makes it so sorting of resultant string works properly).
      Throws:
      ParseException

    • getSecondsSinceEpoch

      public static Date getSecondsSinceEpoch(String timestamp) throws ParseException
      Parameters:
      timestamp - A 14-digit timestamp or the suffix for a 14-digit timestamp: E.g. '20010909014640' or '20010101' or '1970'.
      Returns:
      A date.
      Throws:
      ParseException
      See Also:

    • zeroPadInteger

      public static String zeroPadInteger(int i)
      Parameters:
      i - Integer to add prefix of zeros too. If passed 2005, will return the String 0000002005. String width is the width of Integer.MAX_VALUE as a string (10 digits).
      Returns:
      Padded String version of i.

    • padTo

      public static String padTo(int i, int pad)
      Convert an int to a String, and pad it to pad spaces.
      Parameters:
      i - the int
      pad - the width to pad to.
      Returns:
      String w/ padding.

    • padTo

      public static String padTo(String s, int pad)
      Pad the given String to pad characters wide by pre-pending spaces. s should not be null. If s is already wider than pad no change is done.
      Parameters:
      s - the String to pad
      pad - the width to pad to.
      Returns:
      String w/ padding.

    • padTo

      public static String padTo(String s, int pad, char padChar)
      Pad the given String to pad characters wide by pre-pending padChar. s should not be null. If s is already wider than pad no change is done.
      Parameters:
      s - the String to pad
      pad - the width to pad to.
      padChar - The pad character to use.
      Returns:
      String w/ padding.

    • byteArrayEquals

      public static boolean byteArrayEquals(byte[] lhs, byte[] rhs)
      check that two byte arrays are equal. They may be null.
      Parameters:
      lhs - a byte array
      rhs - another byte array.
      Returns:
      true if they are both equal (or both null)

    • doubleToString

      public static String doubleToString(double val, int maxFractionDigits)
      Converts a double to a string.
      Parameters:
      val - The double to convert
      maxFractionDigits - How many characters to include after '.'
      Returns:
      the double as a string.

    • doubleToString

      public static String doubleToString(double val, int maxFractionDigits, int minFractionDigits)

    • formatBytesForDisplay

      public static String formatBytesForDisplay(long amount)
      Takes a byte size and formats it for display with 'friendly' units.

      This involves converting it to the largest unit (of B, KiB, MiB, GiB, TiB) for which the amount will be > 1.

      Additionally, at least 2 significant digits are always displayed.

      Negative numbers will be returned as '0 B'.

      Parameters:
      amount - the amount of bytes
      Returns:
      A string containing the amount, properly formated.

    • formatMillisecondsToConventional

      public static String formatMillisecondsToConventional(long time)
      Convert milliseconds value to a human-readable duration
      Parameters:
      time -
      Returns:
      Human readable string version of passed time

    • formatMillisecondsToConventional

      public static String formatMillisecondsToConventional(long duration, int unitCount)
      Convert milliseconds value to a human-readable duration of mixed units, using units no larger than days. For example, "5d12h13m12s113ms" or "19h51m".
      Parameters:
      duration -
      unitCount - how many significant units to show, at most for example, a value of 2 would show days+hours or hours+seconds but not hours+second+milliseconds
      Returns:
      Human readable string version of passed time

    • longIntoByteArray

      public static void longIntoByteArray(long l, byte[] array, int offset)
      Copy the raw bytes of a long into a byte array, starting at the specified offset.
      Parameters:
      l -
      array -
      offset -

    • byteArrayIntoLong

      public static long byteArrayIntoLong(byte[] bytearray)

    • byteArrayIntoLong

      public static long byteArrayIntoLong(byte[] bytearray, int offset)
      Byte array into long.
      Parameters:
      bytearray - Array to convert to a long.
      offset - Offset into array at which we start decoding the long.
      Returns:
      Long made of the bytes of array beginning at offset offset.
      See Also:

    • addImpliedHttpIfNecessary

      public static String addImpliedHttpIfNecessary(String u)
      Given a string that may be a plain host or host/path (without URI scheme), add an implied http:// if necessary.
      Parameters:
      u - string to evaluate
      Returns:
      string with http:// added if no scheme already present

    • startsWith

      public static boolean startsWith(byte[] array, byte[] prefix)
      Verify that the array begins with the prefix.
      Parameters:
      array -
      prefix -
      Returns:
      true if array is identical to prefix for the first prefix.length positions

    • prettyString

      public static String prettyString(Object obj)
      Enhance given object's default String display for appearing nested in a pretty Map String.
      Parameters:
      obj - Object to prettify
      Returns:
      prettified String

    • prettyString

      public static String prettyString(Map<?,?> map)
      Provide a improved String of a Map's entries
      Returns:
      prettified (in curly brackets) string of Map contents

    • prettyString

      public static String prettyString(Object[] array)
      Provide a slightly-improved String of Object[]
      Returns:
      prettified (in square brackets) of Object[]

    • isTld

      public static boolean isTld(String dom)
      Return whether the given string represents a known top-level-domain (like "com", "org", etc.) per IANA as of 20100419
      Parameters:
      dom - candidate string
      Returns:
      boolean true if recognized as TLD

    • closeQuietly

      public static void closeQuietly(Object input)

    • continueCheck

      public static void continueCheck() throws InterruptedException
      Perform checks as to whether normal execution should proceed. If an external interrupt is detected, throw an interrupted exception. Used before anything that should not be attempted by a 'zombie' thread that the Frontier/Crawl has given up on.
      Throws:
      InterruptedException

    • readFully

      public static int readFully(InputStream input, byte[] buf) throws IOException
      Read stream into buf until EOF or buf full.
      Parameters:
      input -
      buf -
      Throws:
      IOException

    • getBufferedReader

      public static BufferedReader getBufferedReader(File source) throws IOException
      Get a BufferedReader on the crawler journal given TODO: move to a general utils class
      Parameters:
      source - File journal
      Returns:
      journal buffered reader.
      Throws:
      IOException

    • getBufferedReader

      public static BufferedReader getBufferedReader(URL source) throws IOException
      Get a BufferedReader on the crawler journal given.
      Parameters:
      source - URL journal
      Returns:
      journal buffered reader.
      Throws:
      IOException

    • gzip

      public static byte[] gzip(byte[] bytes) throws IOException
      Gzip passed bytes. Use only when bytes is small.
      Parameters:
      bytes - What to gzip.
      Returns:
      A gzip member of bytes.
      Throws:
      IOException

    • isGzipped

      public static boolean isGzipped(InputStream is) throws IOException
      Tests passed stream is gzip stream by reading in the HEAD. Does not mark/reset stream -- so this test actually makes stream unopenable within GZIP streams, unless reset.
      Parameters:
      is - An InputStream.
      Returns:
      True if compressed stream.
      Throws:
      IOException