org.archive.util

Class ArchiveUtils

java.lang.Object

org.archive.util.ArchiveUtils

public class ArchiveUtils
extends java.lang.Object

Miscellaneous useful methods.

Author:

gojomo & others

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	GZIP_SUFFIX suffix to recognize gzipped files
protected static java.lang.String	LAST_TIMESTAMP14
protected static java.lang.String	LAST_TIMESTAMP17
protected static long	LAST_UNIQUE_NOW14
protected static long	LAST_UNIQUE_NOW17
static int	MAX_INT_CHAR_WIDTH
static java.util.Set<java.lang.String>	TLDS
static java.lang.String	VERSION

Constructor Summary

Constructors

Constructor and Description
ArchiveUtils()

Method Summary

Modifier and Type	Method and Description
static java.lang.String	addImpliedHttpIfNecessary(java.lang.String u) Given a string that may be a plain host or host/path (without URI scheme), add an implied http:// if necessary.
static boolean	byteArrayEquals(byte[] lhs, byte[] rhs) check that two byte arrays are equal.
static long	byteArrayIntoLong(byte[] bytearray)
static long	byteArrayIntoLong(byte[] bytearray, int offset) Byte array into long.
static void	closeQuietly(java.lang.Object input)
static void	continueCheck() Perform checks as to whether normal execution should proceed.
static java.lang.String	dateToTimestamp(java.lang.String input) Convert a user-entered date into a timestamp
static java.lang.String	doubleToString(double val, int maxFractionDigits) Converts a double to a string.
static java.lang.String	doubleToString(double val, int maxFractionDigits, int minFractionDigits)
static java.lang.String	formatBytesForDisplay(long amount) Takes a byte size and formats it for display with 'friendly' units.
static java.lang.String	formatMillisecondsToConventional(long time) Convert milliseconds value to a human-readable duration
static java.lang.String	formatMillisecondsToConventional(long duration, int unitCount) Convert milliseconds value to a human-readable duration of mixed units, using units no larger than days.
static java.lang.String	get12DigitDate() Utility function for creating arc-style date stamps in the format yyyyMMddHHmm.
static java.lang.String	get12DigitDate(java.util.Date d)
static java.lang.String	get12DigitDate(long date) Utility function for creating arc-style date stamps in the format yyyyMMddHHmm.
static java.lang.String	get14DigitDate() Utility function for creating arc-style date stamps in the format yyyyMMddHHmmss.
static java.lang.String	get14DigitDate(java.util.Date d)
static java.lang.String	get14DigitDate(long date) Utility function for creating arc-style date stamps in the format yyyyMMddHHmmss.
static java.lang.String	get17DigitDate() Utility function for creating arc-style date stamps in the format yyyMMddHHmmssSSS.
static java.lang.String	get17DigitDate(java.util.Date date)
static java.lang.String	get17DigitDate(long date) Utility function for creating arc-style date stamps in the format yyyyMMddHHmmssSSS.
static java.io.BufferedReader	getBufferedReader(java.io.File source) Get a BufferedReader on the crawler journal given TODO: move to a general utils class
static java.io.BufferedReader	getBufferedReader(java.net.URL source) Get a BufferedReader on the crawler journal given.
static java.util.Date	getDate(java.lang.String d) Parses an ARC-style date.
static java.util.Date	getDate(java.lang.String d, java.util.Date defaultDate) A version of getDate which returns the default instead of throwing an exception if parsing fails
static java.lang.String	getLog14Date() Utility function for creating log timestamps, in W3C/ISO8601 format, assuming UTC.
static java.lang.String	getLog14Date(java.util.Date date) Utility function for creating log timestamps, in W3C/ISO8601 format, assuming UTC.
static java.lang.String	getLog14Date(long date) Utility function for creating log timestamps, in W3C/ISO8601 format, assuming UTC.
static java.lang.String	getLog17Date() Utility function for creating log timestamps, in W3C/ISO8601 format, assuming UTC.
static java.lang.String	getLog17Date(long date) Utility function for creating log timestamps, in W3C/ISO8601 format, assuming UTC.
static java.util.Date	getSecondsSinceEpoch(java.lang.String timestamp)
static java.lang.String	getUnique14DigitDate() Utility function for creating UNIQUE-from-this-class arc-style date stamps in the format yyyMMddHHmmss.
static java.lang.String	getUnique17DigitDate() Utility function for creating UNIQUE-from-this-class arc-style date stamps in the format yyyMMddHHmmssSSS.
static byte[]	gzip(byte[] bytes) Gzip passed bytes.
static boolean	isGzipped(java.io.InputStream is) Tests passed stream is gzip stream by reading in the HEAD.
static boolean	isTld(java.lang.String dom) Return whether the given string represents a known top-level-domain (like "com", "org", etc.) per IANA as of 20100419
static void	longIntoByteArray(long l, byte[] array, int offset) Copy the raw bytes of a long into a byte array, starting at the specified offset.
static java.lang.String	padTo(int i, int pad) Convert an int to a String, and pad it to pad spaces.
static java.lang.String	padTo(java.lang.String s, int pad) Pad the given String to pad characters wide by pre-pending spaces.
static java.lang.String	padTo(java.lang.String s, int pad, char padChar) Pad the given String to pad characters wide by pre-pending padChar.
static java.util.Date	parse12DigitDate(java.lang.String date) Utility function for parsing arc-style date stamps in the format yyyMMddHHmm.
static java.util.Date	parse14DigitDate(java.lang.String date) Utility function for parsing arc-style date stamps in the format yyyMMddHHmmss.
static java.util.Date	parse14DigitISODate(java.lang.String datetime, java.util.Date defaultVal)
static java.util.Date	parse17DigitDate(java.lang.String date) Utility function for parsing arc-style date stamps in the format yyyMMddHHmmssSSS.
static java.lang.String	prettyString(java.util.Map<?,?> map) Provide a improved String of a Map's entries
static java.lang.String	prettyString(java.lang.Object obj) Enhance given object's default String display for appearing nested in a pretty Map String.
static java.lang.String	prettyString(java.lang.Object[] array) Provide a slightly-improved String of Object[]
static int	readFully(java.io.InputStream input, byte[] buf) Read stream into buf until EOF or buf full.
static java.lang.String	secondsSinceEpoch(java.lang.String timestamp)
static boolean	startsWith(byte[] array, byte[] prefix) Verify that the array begins with the prefix.
static java.lang.String	zeroPadInteger(int i)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

VERSION

public static final java.lang.String VERSION

MAX_INT_CHAR_WIDTH

public static int MAX_INT_CHAR_WIDTH

LAST_UNIQUE_NOW17

protected static long LAST_UNIQUE_NOW17

LAST_TIMESTAMP17

protected static java.lang.String LAST_TIMESTAMP17

LAST_UNIQUE_NOW14

protected static long LAST_UNIQUE_NOW14

LAST_TIMESTAMP14

protected static java.lang.String LAST_TIMESTAMP14

TLDS

public static java.util.Set<java.lang.String> TLDS

GZIP_SUFFIX

public static final java.lang.String GZIP_SUFFIX

suffix to recognize gzipped files

See Also:

Constant Field Values

Constructor Detail

ArchiveUtils

public ArchiveUtils()

Method Detail

get17DigitDate

public static java.lang.String get17DigitDate()

Utility function for creating arc-style date stamps in the format yyyMMddHHmmssSSS. Date stamps are in the UTC time zone

Returns:

the date stamp

getUnique17DigitDate

public static java.lang.String getUnique17DigitDate()

Utility function for creating UNIQUE-from-this-class arc-style date stamps in the format yyyMMddHHmmssSSS. Rather than giving a duplicate datestamp on a subsequent call, will increment the milliseconds until a unique value is returned. Date stamps are in the UTC time zone

Returns:

the date stamp

get14DigitDate

public static java.lang.String get14DigitDate()

Utility function for creating arc-style date stamps in the format yyyyMMddHHmmss. Date stamps are in the UTC time zone

Returns:

the date stamp

getUnique14DigitDate

public static java.lang.String getUnique14DigitDate()

Utility function for creating UNIQUE-from-this-class arc-style date stamps in the format yyyMMddHHmmss. Rather than giving a duplicate datestamp on a subsequent call, will increment the seconds until a unique value is returned. Date stamps are in the UTC time zone

Returns:

the date stamp

get12DigitDate

public static java.lang.String get12DigitDate()

Utility function for creating arc-style date stamps in the format yyyyMMddHHmm. Date stamps are in the UTC time zone

Returns:

the date stamp

getLog17Date

public static java.lang.String getLog17Date()

Utility function for creating log timestamps, in W3C/ISO8601 format, assuming UTC. Use current time. Format is yyyy-MM-dd'T'HH:mm:ss.SSS'Z'

Returns:

the date stamp

getLog17Date

public static java.lang.String getLog17Date(long date)

Utility function for creating log timestamps, in W3C/ISO8601 format, assuming UTC. Format is yyyy-MM-dd'T'HH:mm:ss.SSS'Z'

Parameters:

date - Date to format.

Returns:

the date stamp

getLog14Date

public static java.lang.String getLog14Date()

Utility function for creating log timestamps, in W3C/ISO8601 format, assuming UTC. Use current time. Format is yyyy-MM-dd'T'HH:mm:ss'Z'

Returns:

the date stamp

getLog14Date

public static java.lang.String getLog14Date(long date)

Utility function for creating log timestamps, in W3C/ISO8601 format, assuming UTC. Format is yyyy-MM-dd'T'HH:mm:ss'Z'

Parameters:

date - long timestamp to format.

Returns:

the date stamp

getLog14Date

public static java.lang.String getLog14Date(java.util.Date date)

Utility function for creating log timestamps, in W3C/ISO8601 format, assuming UTC. Format is yyyy-MM-dd'T'HH:mm:ss'Z'

Parameters:

date - Date to format.

Returns:

the date stamp

parse14DigitISODate

public static java.util.Date parse14DigitISODate(java.lang.String datetime,
                                                 java.util.Date defaultVal)

get17DigitDate

public static java.lang.String get17DigitDate(long date)

Utility function for creating arc-style date stamps in the format yyyyMMddHHmmssSSS. Date stamps are in the UTC time zone

Parameters:

date - milliseconds since epoc

Returns:

the date stamp

get17DigitDate

public static java.lang.String get17DigitDate(java.util.Date date)

get14DigitDate

public static java.lang.String get14DigitDate(long date)

Utility function for creating arc-style date stamps in the format yyyyMMddHHmmss. Date stamps are in the UTC time zone

Parameters:

date - milliseconds since epoc

Returns:

the date stamp

get14DigitDate

public static java.lang.String get14DigitDate(java.util.Date d)

get12DigitDate

public static java.lang.String get12DigitDate(long date)

Utility function for creating arc-style date stamps in the format yyyyMMddHHmm. Date stamps are in the UTC time zone

Parameters:

date - milliseconds since epoc

Returns:

the date stamp

get12DigitDate

public static java.lang.String get12DigitDate(java.util.Date d)

getDate

public static java.util.Date getDate(java.lang.String d,
                                     java.util.Date defaultDate)

A version of getDate which returns the default instead of throwing an exception if parsing fails

Parameters:

d -

defaultDate -

Returns:

Throws:

java.text.ParseException

getDate

public static java.util.Date getDate(java.lang.String d)
                              throws java.text.ParseException

Parses an ARC-style date. If passed String is < 12 characters in length, we pad. At a minimum, String should contain a year (>=4 characters). Parse will also fail if day or month are incompletely specified. Depends on the above getXXDigitDate methods.

Parameters:

A - 4-17 digit date in ARC style (yyyy to yyyyMMddHHmmssSSS) formatting.

Returns:

A Date object representing the passed String.

Throws:

java.text.ParseException

dateToTimestamp

public static java.lang.String dateToTimestamp(java.lang.String input)

Convert a user-entered date into a timestamp

Parameters:

input -

Returns:

parse17DigitDate

public static java.util.Date parse17DigitDate(java.lang.String date)
                                       throws java.text.ParseException

Utility function for parsing arc-style date stamps in the format yyyMMddHHmmssSSS. Date stamps are in the UTC time zone. The whole string will not be parsed, only the first 17 digits.

Parameters:

date - an arc-style formatted date stamp

Returns:

the Date corresponding to the date stamp string

Throws:

java.text.ParseException - if the inputstring was malformed

parse14DigitDate

public static java.util.Date parse14DigitDate(java.lang.String date)
                                       throws java.text.ParseException

Utility function for parsing arc-style date stamps in the format yyyMMddHHmmss. Date stamps are in the UTC time zone. The whole string will not be parsed, only the first 14 digits.

Parameters:

date - an arc-style formatted date stamp

Returns:

the Date corresponding to the date stamp string

Throws:

java.text.ParseException - if the inputstring was malformed

parse12DigitDate

public static java.util.Date parse12DigitDate(java.lang.String date)
                                       throws java.text.ParseException

Utility function for parsing arc-style date stamps in the format yyyMMddHHmm. Date stamps are in the UTC time zone. The whole string will not be parsed, only the first 12 digits.

Parameters:

date - an arc-style formatted date stamp

Returns:

the Date corresponding to the date stamp string

Throws:

java.text.ParseException - if the inputstring was malformed

secondsSinceEpoch

public static java.lang.String secondsSinceEpoch(java.lang.String timestamp)
                                          throws java.text.ParseException

Parameters:

timestamp - A 14-digit timestamp or the suffix for a 14-digit timestamp: E.g. '20010909014640' or '20010101' or '1970'.

Returns:

Seconds since the epoch as a string zero-pre-padded so always Integer.MAX_VALUE wide (Makes it so sorting of resultant string works properly).

Throws:

java.text.ParseException

getSecondsSinceEpoch

public static java.util.Date getSecondsSinceEpoch(java.lang.String timestamp)
                                           throws java.text.ParseException

Parameters:

timestamp - A 14-digit timestamp or the suffix for a 14-digit timestamp: E.g. '20010909014640' or '20010101' or '1970'.

Returns:

A date.

Throws:

java.text.ParseException

See Also:

secondsSinceEpoch(String)

zeroPadInteger

public static java.lang.String zeroPadInteger(int i)

Parameters:

i - Integer to add prefix of zeros too. If passed 2005, will return the String 0000002005. String width is the width of Integer.MAX_VALUE as a string (10 digits).

Returns:

Padded String version of i.

padTo

public static java.lang.String padTo(int i,
                                     int pad)

Convert an int to a String, and pad it to pad spaces.

Parameters:

i - the int

pad - the width to pad to.

Returns:

String w/ padding.

padTo

public static java.lang.String padTo(java.lang.String s,
                                     int pad)

Pad the given String to pad characters wide by pre-pending spaces. s should not be null. If s is already wider than pad no change is done.

Parameters:

s - the String to pad

pad - the width to pad to.

Returns:

String w/ padding.

padTo

public static java.lang.String padTo(java.lang.String s,
                                     int pad,
                                     char padChar)

Pad the given String to pad characters wide by pre-pending padChar. s should not be null. If s is already wider than pad no change is done.

Parameters:

s - the String to pad

pad - the width to pad to.

padChar - The pad character to use.

Returns:

String w/ padding.

byteArrayEquals

public static boolean byteArrayEquals(byte[] lhs,
                                      byte[] rhs)

check that two byte arrays are equal. They may be null.

Parameters:

lhs - a byte array

rhs - another byte array.

Returns:

true if they are both equal (or both null)

doubleToString

public static java.lang.String doubleToString(double val,
                                              int maxFractionDigits)

Converts a double to a string.

Parameters:

val - The double to convert

precision - How many characters to include after '.'

Returns:

the double as a string.

doubleToString

public static java.lang.String doubleToString(double val,
                                              int maxFractionDigits,
                                              int minFractionDigits)

formatBytesForDisplay

public static java.lang.String formatBytesForDisplay(long amount)

Takes a byte size and formats it for display with 'friendly' units.

This involves converting it to the largest unit (of B, KiB, MiB, GiB, TiB) for which the amount will be > 1.

Additionally, at least 2 significant digits are always displayed.

Negative numbers will be returned as '0 B'.

Parameters:

amount - the amount of bytes

Returns:

A string containing the amount, properly formated.

formatMillisecondsToConventional

public static java.lang.String formatMillisecondsToConventional(long time)

Convert milliseconds value to a human-readable duration

Parameters:

time -

Returns:

Human readable string version of passed time

formatMillisecondsToConventional

public static java.lang.String formatMillisecondsToConventional(long duration,
                                                                int unitCount)

Convert milliseconds value to a human-readable duration of mixed units, using units no larger than days. For example, "5d12h13m12s113ms" or "19h51m".

Parameters:

duration -

unitCount - how many significant units to show, at most for example, a value of 2 would show days+hours or hours+seconds but not hours+second+milliseconds

Returns:

Human readable string version of passed time

longIntoByteArray

public static void longIntoByteArray(long l,
                                     byte[] array,
                                     int offset)

Copy the raw bytes of a long into a byte array, starting at the specified offset.

Parameters:

l -

array -

offset -

byteArrayIntoLong

public static long byteArrayIntoLong(byte[] bytearray)

byteArrayIntoLong

public static long byteArrayIntoLong(byte[] bytearray,
                                     int offset)

Byte array into long.

Parameters:

bytearray - Array to convert to a long.

offset - Offset into array at which we start decoding the long.

Returns:

Long made of the bytes of array beginning at offset offset.

See Also:

longIntoByteArray(long, byte[], int)

addImpliedHttpIfNecessary

public static java.lang.String addImpliedHttpIfNecessary(java.lang.String u)

Given a string that may be a plain host or host/path (without URI scheme), add an implied http:// if necessary.

Parameters:

u - string to evaluate

Returns:

string with http:// added if no scheme already present

startsWith

public static boolean startsWith(byte[] array,
                                 byte[] prefix)

Verify that the array begins with the prefix.

Parameters:

array -

prefix -

Returns:

true if array is identical to prefix for the first prefix.length positions

prettyString

public static java.lang.String prettyString(java.lang.Object obj)

Enhance given object's default String display for appearing nested in a pretty Map String.

Parameters:

obj - Object to prettify

Returns:

prettified String

prettyString

public static java.lang.String prettyString(java.util.Map<?,?> map)

Provide a improved String of a Map's entries

Parameters:

Map -

Returns:

prettified (in curly brackets) string of Map contents

prettyString

public static java.lang.String prettyString(java.lang.Object[] array)

Provide a slightly-improved String of Object[]

Parameters:

Object[] -

Returns:

prettified (in square brackets) of Object[]

isTld

public static boolean isTld(java.lang.String dom)

Return whether the given string represents a known top-level-domain (like "com", "org", etc.) per IANA as of 20100419

Parameters:

dom - candidate string

Returns:

boolean true if recognized as TLD

closeQuietly

public static void closeQuietly(java.lang.Object input)

continueCheck

public static void continueCheck()
                          throws java.lang.InterruptedException

Perform checks as to whether normal execution should proceed. If an external interrupt is detected, throw an interrupted exception. Used before anything that should not be attempted by a 'zombie' thread that the Frontier/Crawl has given up on.

Throws:

java.lang.InterruptedException

readFully

public static int readFully(java.io.InputStream input,
                            byte[] buf)
                     throws java.io.IOException

Read stream into buf until EOF or buf full.

Parameters:

input -

buf -

Throws:

java.io.IOException

getBufferedReader

public static java.io.BufferedReader getBufferedReader(java.io.File source)
                                                throws java.io.IOException

Get a BufferedReader on the crawler journal given TODO: move to a general utils class

Parameters:

source - File journal

Returns:

journal buffered reader.

Throws:

java.io.IOException

getBufferedReader

public static java.io.BufferedReader getBufferedReader(java.net.URL source)
                                                throws java.io.IOException

Get a BufferedReader on the crawler journal given.

Parameters:

source - URL journal

Returns:

journal buffered reader.

Throws:

java.io.IOException

gzip

public static byte[] gzip(byte[] bytes)
                   throws java.io.IOException

Gzip passed bytes. Use only when bytes is small.

Parameters:

bytes - What to gzip.

Returns:

A gzip member of bytes.

Throws:

java.io.IOException

isGzipped

public static boolean isGzipped(java.io.InputStream is)
                         throws java.io.IOException

Tests passed stream is gzip stream by reading in the HEAD. Does not mark/reset stream -- so this test actually makes stream unopenable within GZIP streams, unless reset.

Parameters:

is - An InputStream.

Returns:

True if compressed stream.

Throws:

java.io.IOException