org.archive.io.arc

Class ARCRecordMetaData

java.lang.Object

org.archive.io.arc.ARCRecordMetaData

All Implemented Interfaces:

ARCConstants, ArchiveFileConstants, ArchiveRecordHeader

public class ARCRecordMetaData
extends Object
implements ArchiveRecordHeader, ARCConstants

An immutable class to hold an ARC record meta data.

Author:

stack

Nested Class Summary

Nested classes/interfaces inherited from interface org.archive.format.arc.ARCConstants

ARCConstants.ArcRecordErrors

Field Summary

Fields

Modifier and Type	Field and Description
protected Map<String,Object>	headerFields Map of record header fields.

Fields inherited from interface org.archive.format.arc.ARCConstants

ACTUAL_LENGTH_KEY, ALEXA_DAT_MIME, ARC_FILE_EXTENSION, ARC_GZIP_EXTRA_FIELD, ARC_MAGIC_NUMBER, ARC_META_CHARSET, CARRIAGE_RETURN_ORD, CHECKSUM_FIELD_KEY, CHECKSUM_HEADER_FIELD_KEY, CODE_HEADER_FIELD_KEY, COMPRESSED_ARC_FILE_EXTENSION, DATE_STRING_KEY, DECLARED_LENGTH_KEY, DEFAULT_ENCODING, DEFAULT_GZIP_HEADER_LENGTH, DEFAULT_MAX_ARC_FILE_SIZE, DEFAULT_MIME, DELIMITER, DNS_MIME, DOT_ARC_FILE_EXTENSION, DOT_COMPRESSED_ARC_FILE_EXTENSION, DOT_COMPRESSED_FILE_EXTENSION, FIELD_COUNT, FILEDESC_SCHEME, FILENAME_FIELD_KEY, FILENAME_HEADER_FIELD_KEY, GZIP_HEADER_BEGIN, HEADER_FIELD_SEPARATOR, HEADER_LENGTH, IP_HEADER_FIELD_KEY, IP_KEY, LEADING_NEWLINES_KEY, LINE_SEPARATOR, LOCATION_HEADER_FIELD_KEY, MAX_HEADER_LINE_LENGTH, MAX_META_LENGTH, MAX_METADATA_LINE_LENGTH, MIME_KEY, MINIMUM_RECORD_LENGTH, NEW_LINE_ORD, OFFSET_FIELD_KEY, OFFSET_HEADER_FIELD_KEY, REQUIRED_VERSION_1_HEADER_FIELDS, STATUSCODE_FIELD_KEY, TOKENIZED_PREFIX, TRAILING_CRUFT_KEY, TRAILING_NEWLINES_KEY, URL_KEY

Fields inherited from interface org.archive.format.ArchiveFileConstants

ABSOLUTE_OFFSET_KEY, CDX, CDX_FILE, CDX_LINE_BUFFER_SIZE, CRLF, DATE_FIELD_KEY, DEFAULT_DIGEST_METHOD, DUMP, GZIP_DUMP, HEADER, INVALID_SUFFIX, LENGTH_FIELD_KEY, MIMETYPE_FIELD_KEY, NOHEAD, OCCUPIED_SUFFIX, READER_IDENTIFIER_FIELD_KEY, RECORD_IDENTIFIER_FIELD_KEY, SINGLE_SPACE, TYPE_FIELD_KEY, URL_FIELD_KEY, VERSION_FIELD_KEY

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	ARCRecordMetaData() Shut down the default constructor.
	ARCRecordMetaData(String arc, Map<String,Object> headerFields) Constructor.

Method Summary

Methods

Modifier and Type	Method and Description
String	getArc()
File	getArcFile()
int	getContentBegin() Offset at which the content begins.
long	getContentLength()
String	getDate() Get the time when the record was harvested.
String	getDigest()
Set<String>	getHeaderFieldKeys()
Map<String,Object>	getHeaderFields()
Object	getHeaderValue(String key)
String	getIp()
long	getLength()
String	getMimetype()
long	getOffset()
String	getReaderIdentifier()
String	getRecordIdentifier()
String	getStatusCode()
String	getUrl()
String	getVersion()
protected void	setContentBegin(int offset)
void	setDigest(String d)
void	setStatusCode(String statusCode)
protected void	testRequiredField(Map<String,Object> fields, String requiredField) Test required field is present in hash.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

headerFields

protected Map<String,Object> headerFields

Map of record header fields. We store all in a hashmap. This way we can hold version 1 or version 2 record meta data.

Keys are lowercase.

Constructor Detail

ARCRecordMetaData

protected ARCRecordMetaData()

Shut down the default constructor.

ARCRecordMetaData

public ARCRecordMetaData(String arc,
                 Map<String,Object> headerFields)
                  throws IOException

Constructor.

Parameters:

arc - The arc file this metadata came out of.

headerFields - Hash of meta fields.

Throws:

IOException

Method Detail

testRequiredField

protected void testRequiredField(Map<String,Object> fields,
                     String requiredField)
                          throws IOException

Test required field is present in hash.

Parameters:

fields - Map of fields.

requiredField - Field to test for.

Throws:

IOException - If required field is not present.

getDate

public String getDate()

Get the time when the record was harvested.

Returns the date in Heritrix 14 digit time format (UTC). See the ArchiveUtils class for converting to Java dates.

Specified by:

getDate in interface ArchiveRecordHeader

Returns:

Header date in Heritrix 14 digit format.

See Also:

ArchiveUtils.parse14DigitDate(String)

getLength

public long getLength()

Specified by:

getLength in interface ArchiveRecordHeader

Returns:

Return length of the record.

getContentLength

public long getContentLength()

Specified by:

getContentLength in interface ArchiveRecordHeader

Returns:

Return Content-Length of the contents of the record Same as record length for arcs? TODO

getUrl

public String getUrl()

Specified by:

getUrl in interface ArchiveRecordHeader

Returns:

Header url.

getIp

public String getIp()

Returns:

IP.

getMimetype

public String getMimetype()

Specified by:

getMimetype in interface ArchiveRecordHeader

Returns:

mimetype The mimetype that is in the ARC metaline -- NOT the http content-type content.

getVersion

public String getVersion()

Specified by:

getVersion in interface ArchiveRecordHeader

Returns:

Arcfile version.

getOffset

public long getOffset()

Specified by:

getOffset in interface ArchiveRecordHeader

Returns:

Offset into arcfile at which this record begins.

getHeaderValue

public Object getHeaderValue(String key)

Specified by:

getHeaderValue in interface ArchiveRecordHeader

Parameters:

key - Key to use looking up field value.

Returns:

value for passed key of null if no such entry.

getHeaderFieldKeys

public Set<String> getHeaderFieldKeys()

Specified by:

getHeaderFieldKeys in interface ArchiveRecordHeader

Returns:

Header field name keys.

getHeaderFields

public Map<String,Object> getHeaderFields()

Specified by:

getHeaderFields in interface ArchiveRecordHeader

Returns:

Map of header fields.

getArc

public String getArc()

Returns:

Returns identifier for ARC.

getArcFile

public File getArcFile()

Returns:

Convenience method that does a return new File(this.arc) (Be aware this.arc is not always full path to an ARC file -- may be an URL). Test returned file for existence.

getDigest

public String getDigest()

Specified by:

getDigest in interface ArchiveRecordHeader

Returns:

Returns the digest.

setDigest

public void setDigest(String d)

Parameters:

d - The digest to set.

getStatusCode

public String getStatusCode()

Returns:

Returns the statusCode. May be null.

setStatusCode

public void setStatusCode(String statusCode)

Parameters:

statusCode - The statusCode to set.

toString

public String toString()

Specified by:

toString in interface ArchiveRecordHeader

Overrides:

toString in class Object

getReaderIdentifier

public String getReaderIdentifier()

Specified by:

getReaderIdentifier in interface ArchiveRecordHeader

Returns:

Returns identifier for current Archive file. Be aware this may not be a file name or file path. It may just be an URL. Depends on how Archive file was made.

getRecordIdentifier

public String getRecordIdentifier()

Specified by:

getRecordIdentifier in interface ArchiveRecordHeader

Returns:

Identifier for the record. If ARC, the URL + date. If WARC, the GUID assigned.

getContentBegin

public int getContentBegin()

Description copied from interface: ArchiveRecordHeader

Offset at which the content begins. For ARCs, its used to delimit where http headers end and content begins. For WARCs, its end of Named Fields before payload starts.

Specified by:

getContentBegin in interface ArchiveRecordHeader

setContentBegin

protected void setContentBegin(int offset)