ArchiveUtils (webarchive-commons 1.1.5 API)

java.lang.Object
- org.archive.util.ArchiveUtils

```
public class ArchiveUtils
extends Object
```
Miscellaneous useful methods.

Author:

gojomo & others

Field Summary

Fields
Modifier and Type	Field and Description
`static String`	`GZIP_SUFFIX` suffix to recognize gzipped files
`protected static String`	`LAST_TIMESTAMP14`
`protected static String`	`LAST_TIMESTAMP17`
`protected static long`	`LAST_UNIQUE_NOW14`
`protected static long`	`LAST_UNIQUE_NOW17`
`static int`	`MAX_INT_CHAR_WIDTH`
`static Set<String>`	`TLDS`
`static String`	`VERSION`

Constructor Summary

Constructors
Constructor and Description

ArchiveUtils()

Constructors
Constructor and Description
`ArchiveUtils()`

Method Summary

Methods
Modifier and Type	Method and Description
`static String`	`addImpliedHttpIfNecessary(String u)` Given a string that may be a plain host or host/path (without URI scheme), add an implied http:// if necessary.
`static boolean`	`byteArrayEquals(byte[] lhs, byte[] rhs)` check that two byte arrays are equal.
`static long`	`byteArrayIntoLong(byte[] bytearray)`
`static long`	`byteArrayIntoLong(byte[] bytearray, int offset)` Byte array into long.
`static void`	`closeQuietly(Object input)`
`static void`	`continueCheck()` Perform checks as to whether normal execution should proceed.
`static String`	`dateToTimestamp(String input)` Convert a user-entered date into a timestamp
`static String`	`doubleToString(double val, int maxFractionDigits)` Converts a double to a string.
`static String`	`doubleToString(double val, int maxFractionDigits, int minFractionDigits)`
`static String`	`formatBytesForDisplay(long amount)` Takes a byte size and formats it for display with 'friendly' units.
`static String`	`formatMillisecondsToConventional(long time)` Convert milliseconds value to a human-readable duration
`static String`	`formatMillisecondsToConventional(long duration, int unitCount)` Convert milliseconds value to a human-readable duration of mixed units, using units no larger than days.
`static String`	`get12DigitDate()` Utility function for creating arc-style date stamps in the format yyyyMMddHHmm.
`static String`	`get12DigitDate(Date d)`
`static String`	`get12DigitDate(long date)` Utility function for creating arc-style date stamps in the format yyyyMMddHHmm.
`static String`	`get14DigitDate()` Utility function for creating arc-style date stamps in the format yyyyMMddHHmmss.
`static String`	`get14DigitDate(Date d)`
`static String`	`get14DigitDate(long date)` Utility function for creating arc-style date stamps in the format yyyyMMddHHmmss.
`static String`	`get17DigitDate()` Utility function for creating arc-style date stamps in the format yyyMMddHHmmssSSS.
`static String`	`get17DigitDate(Date date)`
`static String`	`get17DigitDate(long date)` Utility function for creating arc-style date stamps in the format yyyyMMddHHmmssSSS.
`static BufferedReader`	`getBufferedReader(File source)` Get a BufferedReader on the crawler journal given TODO: move to a general utils class
`static BufferedReader`	`getBufferedReader(URL source)` Get a BufferedReader on the crawler journal given.
`static Date`	`getDate(String d)` Parses an ARC-style date.
`static Date`	`getDate(String d, Date defaultDate)` A version of getDate which returns the default instead of throwing an exception if parsing fails
`static String`	`getLog14Date()` Utility function for creating log timestamps, in W3C/ISO8601 format, assuming UTC.
`static String`	`getLog14Date(Date date)` Utility function for creating log timestamps, in W3C/ISO8601 format, assuming UTC.
`static String`	`getLog14Date(long date)` Utility function for creating log timestamps, in W3C/ISO8601 format, assuming UTC.
`static String`	`getLog17Date()` Utility function for creating log timestamps, in W3C/ISO8601 format, assuming UTC.
`static String`	`getLog17Date(long date)` Utility function for creating log timestamps, in W3C/ISO8601 format, assuming UTC.
`static Date`	`getSecondsSinceEpoch(String timestamp)`
`static String`	`getUnique14DigitDate()` Utility function for creating UNIQUE-from-this-class arc-style date stamps in the format yyyMMddHHmmss.
`static String`	`getUnique17DigitDate()` Utility function for creating UNIQUE-from-this-class arc-style date stamps in the format yyyMMddHHmmssSSS.
`static byte[]`	`gzip(byte[] bytes)` Gzip passed bytes.
`static boolean`	`isGzipped(InputStream is)` Tests passed stream is gzip stream by reading in the HEAD.
`static boolean`	`isTld(String dom)` Return whether the given string represents a known top-level-domain (like "com", "org", etc.) per IANA as of 20100419
`static void`	`longIntoByteArray(long l, byte[] array, int offset)` Copy the raw bytes of a long into a byte array, starting at the specified offset.
`static String`	`padTo(int i, int pad)` Convert an `int` to a `String`, and pad it to `pad` spaces.
`static String`	`padTo(String s, int pad)` Pad the given `String` to `pad` characters wide by pre-pending spaces.
`static String`	`padTo(String s, int pad, char padChar)` Pad the given `String` to `pad` characters wide by pre-pending `padChar`.
`static Date`	`parse12DigitDate(String date)` Utility function for parsing arc-style date stamps in the format yyyMMddHHmm.
`static Date`	`parse14DigitDate(String date)` Utility function for parsing arc-style date stamps in the format yyyMMddHHmmss.
`static Date`	`parse14DigitISODate(String datetime, Date defaultVal)`
`static Date`	`parse17DigitDate(String date)` Utility function for parsing arc-style date stamps in the format yyyMMddHHmmssSSS.
`static String`	`prettyString(Map<?,?> map)` Provide a improved String of a Map's entries
`static String`	`prettyString(Object obj)` Enhance given object's default String display for appearing nested in a pretty Map String.
`static String`	`prettyString(Object[] array)` Provide a slightly-improved String of Object[]
`static int`	`readFully(InputStream input, byte[] buf)` Read stream into buf until EOF or buf full.
`static String`	`secondsSinceEpoch(String timestamp)`
`static boolean`	`startsWith(byte[] array, byte[] prefix)` Verify that the array begins with the prefix.
`static String`	`zeroPadInteger(int i)`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - VERSION
```
public static final String VERSION
```
  - MAX_INT_CHAR_WIDTH
```
public static int MAX_INT_CHAR_WIDTH
```
  - LAST_UNIQUE_NOW17
```
protected static long LAST_UNIQUE_NOW17
```
  - LAST_TIMESTAMP17
```
protected static String LAST_TIMESTAMP17
```
  - LAST_UNIQUE_NOW14
```
protected static long LAST_UNIQUE_NOW14
```
  - LAST_TIMESTAMP14
```
protected static String LAST_TIMESTAMP14
```
  - TLDS
```
public static Set<String> TLDS
```
  - GZIP_SUFFIX
```
public static final String GZIP_SUFFIX
```
    suffix to recognize gzipped files
    
    See Also:
    Constant Field Values
- Constructor Detail
  - ArchiveUtils
```
public ArchiveUtils()
```
- Method Detail
  - get17DigitDate
```
public static String get17DigitDate()
```
    Utility function for creating arc-style date stamps in the format yyyMMddHHmmssSSS. Date stamps are in the UTC time zone
    
    Returns:
    the date stamp
  - getUnique17DigitDate
```
public static String getUnique17DigitDate()
```
    Utility function for creating UNIQUE-from-this-class arc-style date stamps in the format yyyMMddHHmmssSSS. Rather than giving a duplicate datestamp on a subsequent call, will increment the milliseconds until a unique value is returned. Date stamps are in the UTC time zone
    
    Returns:
    the date stamp
  - get14DigitDate
```
public static String get14DigitDate()
```
    Utility function for creating arc-style date stamps in the format yyyyMMddHHmmss. Date stamps are in the UTC time zone
    
    Returns:
    the date stamp
  - getUnique14DigitDate
```
public static String getUnique14DigitDate()
```
    Utility function for creating UNIQUE-from-this-class arc-style date stamps in the format yyyMMddHHmmss. Rather than giving a duplicate datestamp on a subsequent call, will increment the seconds until a unique value is returned. Date stamps are in the UTC time zone
    
    Returns:
    the date stamp
  - get12DigitDate
```
public static String get12DigitDate()
```
    Utility function for creating arc-style date stamps in the format yyyyMMddHHmm. Date stamps are in the UTC time zone
    
    Returns:
    the date stamp
  - getLog17Date
```
public static String getLog17Date()
```
    Utility function for creating log timestamps, in W3C/ISO8601 format, assuming UTC. Use current time. Format is yyyy-MM-dd'T'HH:mm:ss.SSS'Z'
    
    Returns:
    the date stamp
  - getLog17Date
```
public static String getLog17Date(long date)
```
    Utility function for creating log timestamps, in W3C/ISO8601 format, assuming UTC. Format is yyyy-MM-dd'T'HH:mm:ss.SSS'Z'
    
    Parameters:
    date - Date to format.
    
    Returns:
    the date stamp
  - getLog14Date
```
public static String getLog14Date()
```
    Utility function for creating log timestamps, in W3C/ISO8601 format, assuming UTC. Use current time. Format is yyyy-MM-dd'T'HH:mm:ss'Z'
    
    Returns:
    the date stamp
  - getLog14Date
```
public static String getLog14Date(long date)
```
    Utility function for creating log timestamps, in W3C/ISO8601 format, assuming UTC. Format is yyyy-MM-dd'T'HH:mm:ss'Z'
    
    Parameters:
    date - long timestamp to format.
    
    Returns:
    the date stamp
  - getLog14Date
```
public static String getLog14Date(Date date)
```
    Utility function for creating log timestamps, in W3C/ISO8601 format, assuming UTC. Format is yyyy-MM-dd'T'HH:mm:ss'Z'
    
    Parameters:
    date - Date to format.
    
    Returns:
    the date stamp
  - parse14DigitISODate
```
public static Date parse14DigitISODate(String datetime,
                       Date defaultVal)
```
  - get17DigitDate
```
public static String get17DigitDate(long date)
```
    Utility function for creating arc-style date stamps in the format yyyyMMddHHmmssSSS. Date stamps are in the UTC time zone
    
    Parameters:
    date - milliseconds since epoc
    
    Returns:
    the date stamp
  - get17DigitDate
```
public static String get17DigitDate(Date date)
```
  - get14DigitDate
```
public static String get14DigitDate(long date)
```
    Utility function for creating arc-style date stamps in the format yyyyMMddHHmmss. Date stamps are in the UTC time zone
    
    Parameters:
    date - milliseconds since epoc
    
    Returns:
    the date stamp
  - get14DigitDate
```
public static String get14DigitDate(Date d)
```
  - get12DigitDate
```
public static String get12DigitDate(long date)
```
    Utility function for creating arc-style date stamps in the format yyyyMMddHHmm. Date stamps are in the UTC time zone
    
    Parameters:
    date - milliseconds since epoc
    
    Returns:
    the date stamp
  - get12DigitDate
```
public static String get12DigitDate(Date d)
```
  - getDate
```
public static Date getDate(String d,
           Date defaultDate)
```
    A version of getDate which returns the default instead of throwing an exception if parsing fails
    
    Parameters:
    d -
    defaultDate -
    
    Returns:
    
    Throws:
    
    ParseException
  - getDate
```
public static Date getDate(String d)
                    throws ParseException
```
    Parses an ARC-style date. If passed String is < 12 characters in length, we pad. At a minimum, String should contain a year (>=4 characters). Parse will also fail if day or month are incompletely specified. Depends on the above getXXDigitDate methods.
    
    Parameters:
    A - 4-17 digit date in ARC style (yyyy to yyyyMMddHHmmssSSS) formatting.
    
    Returns:
    A Date object representing the passed String.
    
    Throws:
    
    ParseException
  - dateToTimestamp
```
public static String dateToTimestamp(String input)
```
    Convert a user-entered date into a timestamp
    
    Parameters:
    input -
    
    Returns:
  - parse17DigitDate
```
public static Date parse17DigitDate(String date)
                             throws ParseException
```
    Utility function for parsing arc-style date stamps in the format yyyMMddHHmmssSSS. Date stamps are in the UTC time zone. The whole string will not be parsed, only the first 17 digits.
    
    Parameters:
    date - an arc-style formatted date stamp
    
    Returns:
    the Date corresponding to the date stamp string
    
    Throws:
    
    ParseException - if the inputstring was malformed
  - parse14DigitDate
```
public static Date parse14DigitDate(String date)
                             throws ParseException
```
    Utility function for parsing arc-style date stamps in the format yyyMMddHHmmss. Date stamps are in the UTC time zone. The whole string will not be parsed, only the first 14 digits.
    
    Parameters:
    date - an arc-style formatted date stamp
    
    Returns:
    the Date corresponding to the date stamp string
    
    Throws:
    
    ParseException - if the inputstring was malformed
  - parse12DigitDate
```
public static Date parse12DigitDate(String date)
                             throws ParseException
```
    Utility function for parsing arc-style date stamps in the format yyyMMddHHmm. Date stamps are in the UTC time zone. The whole string will not be parsed, only the first 12 digits.
    
    Parameters:
    date - an arc-style formatted date stamp
    
    Returns:
    the Date corresponding to the date stamp string
    
    Throws:
    
    ParseException - if the inputstring was malformed
  - secondsSinceEpoch
```
public static String secondsSinceEpoch(String timestamp)
                                throws ParseException
```
    Parameters:
    timestamp - A 14-digit timestamp or the suffix for a 14-digit timestamp: E.g. '20010909014640' or '20010101' or '1970'.
    
    Returns:
    Seconds since the epoch as a string zero-pre-padded so always Integer.MAX_VALUE wide (Makes it so sorting of resultant string works properly).
    
    Throws:
    
    ParseException
  - getSecondsSinceEpoch
```
public static Date getSecondsSinceEpoch(String timestamp)
                                 throws ParseException
```
    Parameters:
    timestamp - A 14-digit timestamp or the suffix for a 14-digit timestamp: E.g. '20010909014640' or '20010101' or '1970'.
    
    Returns:
    A date.
    
    Throws:
    
    ParseException
    See Also:
    secondsSinceEpoch(String)
  - zeroPadInteger
```
public static String zeroPadInteger(int i)
```
    Parameters:
    i - Integer to add prefix of zeros too. If passed 2005, will return the String 0000002005. String width is the width of Integer.MAX_VALUE as a string (10 digits).
    
    Returns:
    Padded String version of i.
  - padTo
```
public static String padTo(int i,
           int pad)
```
    Convert an int to a String, and pad it to pad spaces.
    
    Parameters:
    i - the int
    pad - the width to pad to.
    
    Returns:
    String w/ padding.
  - padTo
```
public static String padTo(String s,
           int pad)
```
    Pad the given String to pad characters wide by pre-pending spaces. s should not be null. If s is already wider than pad no change is done.
    
    Parameters:
    s - the String to pad
    pad - the width to pad to.
    
    Returns:
    String w/ padding.
  - padTo
```
public static String padTo(String s,
           int pad,
           char padChar)
```
    Pad the given String to pad characters wide by pre-pending padChar. s should not be null. If s is already wider than pad no change is done.
    
    Parameters:
    s - the String to pad
    pad - the width to pad to.
    padChar - The pad character to use.
    
    Returns:
    String w/ padding.
  - byteArrayEquals
```
public static boolean byteArrayEquals(byte[] lhs,
                      byte[] rhs)
```
    check that two byte arrays are equal. They may be null.
    
    Parameters:
    lhs - a byte array
    rhs - another byte array.
    
    Returns:
    true if they are both equal (or both null)
  - doubleToString
```
public static String doubleToString(double val,
                    int maxFractionDigits)
```
    Converts a double to a string.
    
    Parameters:
    val - The double to convert
    precision - How many characters to include after '.'
    
    Returns:
    the double as a string.
  - doubleToString
```
public static String doubleToString(double val,
                    int maxFractionDigits,
                    int minFractionDigits)
```
  - formatBytesForDisplay
```
public static String formatBytesForDisplay(long amount)
```
    Takes a byte size and formats it for display with 'friendly' units.
    This involves converting it to the largest unit (of B, KiB, MiB, GiB, TiB) for which the amount will be > 1.
    Additionally, at least 2 significant digits are always displayed.
    Negative numbers will be returned as '0 B'.
    
    Parameters:
    amount - the amount of bytes
    
    Returns:
    A string containing the amount, properly formated.
  - formatMillisecondsToConventional
```
public static String formatMillisecondsToConventional(long time)
```
    Convert milliseconds value to a human-readable duration
    
    Parameters:
    time -
    
    Returns:
    Human readable string version of passed time
  - formatMillisecondsToConventional
```
public static String formatMillisecondsToConventional(long duration,
                                      int unitCount)
```
    Convert milliseconds value to a human-readable duration of mixed units, using units no larger than days. For example, "5d12h13m12s113ms" or "19h51m".
    
    Parameters:
    duration -
    unitCount - how many significant units to show, at most for example, a value of 2 would show days+hours or hours+seconds but not hours+second+milliseconds
    
    Returns:
    Human readable string version of passed time
  - longIntoByteArray
```
public static void longIntoByteArray(long l,
                     byte[] array,
                     int offset)
```
    Copy the raw bytes of a long into a byte array, starting at the specified offset.
    
    Parameters:
    l -
    array -
    offset -
  - byteArrayIntoLong
```
public static long byteArrayIntoLong(byte[] bytearray)
```
  - byteArrayIntoLong
```
public static long byteArrayIntoLong(byte[] bytearray,
                     int offset)
```
    Byte array into long.
    
    Parameters:
    bytearray - Array to convert to a long.
    offset - Offset into array at which we start decoding the long.
    
    Returns:
    Long made of the bytes of array beginning at offset offset.
    See Also:
    longIntoByteArray(long, byte[], int)
  - addImpliedHttpIfNecessary
```
public static String addImpliedHttpIfNecessary(String u)
```
    Given a string that may be a plain host or host/path (without URI scheme), add an implied http:// if necessary.
    
    Parameters:
    u - string to evaluate
    
    Returns:
    string with http:// added if no scheme already present
  - startsWith
```
public static boolean startsWith(byte[] array,
                 byte[] prefix)
```
    Verify that the array begins with the prefix.
    
    Parameters:
    array -
    prefix -
    
    Returns:
    true if array is identical to prefix for the first prefix.length positions
  - prettyString
```
public static String prettyString(Object obj)
```
    Enhance given object's default String display for appearing nested in a pretty Map String.
    
    Parameters:
    obj - Object to prettify
    
    Returns:
    prettified String
  - prettyString
```
public static String prettyString(Map<?,?> map)
```
    Provide a improved String of a Map's entries
    
    Parameters:
    Map -
    
    Returns:
    prettified (in curly brackets) string of Map contents
  - prettyString
```
public static String prettyString(Object[] array)
```
    Provide a slightly-improved String of Object[]
    
    Parameters:
    Object[] -
    
    Returns:
    prettified (in square brackets) of Object[]
  - isTld
```
public static boolean isTld(String dom)
```
    Return whether the given string represents a known top-level-domain (like "com", "org", etc.) per IANA as of 20100419
    
    Parameters:
    dom - candidate string
    
    Returns:
    boolean true if recognized as TLD
  - closeQuietly
```
public static void closeQuietly(Object input)
```
  - continueCheck
```
public static void continueCheck()
                          throws InterruptedException
```
    Perform checks as to whether normal execution should proceed. If an external interrupt is detected, throw an interrupted exception. Used before anything that should not be attempted by a 'zombie' thread that the Frontier/Crawl has given up on.
    
    Throws:
    
    InterruptedException
  - readFully
```
public static int readFully(InputStream input,
            byte[] buf)
                     throws IOException
```
    Read stream into buf until EOF or buf full.
    
    Parameters:
    input -
    buf -
    
    Throws:
    
    IOException
  - getBufferedReader
```
public static BufferedReader getBufferedReader(File source)
                                        throws IOException
```
    Get a BufferedReader on the crawler journal given TODO: move to a general utils class
    
    Parameters:
    source - File journal
    
    Returns:
    journal buffered reader.
    
    Throws:
    
    IOException
  - getBufferedReader
```
public static BufferedReader getBufferedReader(URL source)
                                        throws IOException
```
    Get a BufferedReader on the crawler journal given.
    
    Parameters:
    source - URL journal
    
    Returns:
    journal buffered reader.
    
    Throws:
    
    IOException
  - gzip
```
public static byte[] gzip(byte[] bytes)
                   throws IOException
```
    Gzip passed bytes. Use only when bytes is small.
    
    Parameters:
    bytes - What to gzip.
    
    Returns:
    A gzip member of bytes.
    
    Throws:
    
    IOException
  - isGzipped
```
public static boolean isGzipped(InputStream is)
                         throws IOException
```
    Tests passed stream is gzip stream by reading in the HEAD. Does not mark/reset stream -- so this test actually makes stream unopenable within GZIP streams, unless reset.
    
    Parameters:
    is - An InputStream.
    
    Returns:
    True if compressed stream.
    
    Throws:
    
    IOException

Class ArchiveUtils

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

VERSION

MAX_INT_CHAR_WIDTH

LAST_UNIQUE_NOW17

LAST_TIMESTAMP17

LAST_UNIQUE_NOW14

LAST_TIMESTAMP14

TLDS

GZIP_SUFFIX

Constructor Detail

ArchiveUtils

Method Detail

get17DigitDate

getUnique17DigitDate

get14DigitDate

getUnique14DigitDate

get12DigitDate

getLog17Date

getLog17Date

getLog14Date

getLog14Date

getLog14Date

parse14DigitISODate

get17DigitDate

get17DigitDate

get14DigitDate

get14DigitDate

get12DigitDate

get12DigitDate

getDate

getDate

dateToTimestamp

parse17DigitDate

parse14DigitDate

parse12DigitDate

secondsSinceEpoch

getSecondsSinceEpoch

zeroPadInteger

padTo

padTo

padTo

byteArrayEquals

doubleToString

doubleToString

formatBytesForDisplay

formatMillisecondsToConventional

formatMillisecondsToConventional

longIntoByteArray

byteArrayIntoLong

byteArrayIntoLong

addImpliedHttpIfNecessary

startsWith

prettyString

prettyString

prettyString

isTld

closeQuietly

continueCheck

readFully

getBufferedReader

getBufferedReader

gzip

isGzipped