org.archive.io.arc

Class ARCRecord

java.lang.Object

java.io.InputStream

org.archive.io.ArchiveRecord

org.archive.io.arc.ARCRecord

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, ARCConstants, ArchiveFileConstants

public class ARCRecord
extends ArchiveRecord
implements ARCConstants

An ARC file record. Does not compass the ARCRecord metadata line, just the record content.

Author:

stack

Nested Class Summary

Nested classes/interfaces inherited from interface org.archive.format.arc.ARCConstants

ARCConstants.ArcRecordErrors

Field Summary

Fields

Modifier and Type	Field and Description
long	compressedBytes null if source was not compressed
java.util.List<ARCConstants.ArcRecordErrors>	errors errors encountered reading record
long	httpHeaderBytesRead Http header bytes read while trying to read http header
long	httpPayloadActualLength actual http payload length, should match http-payload-declared-length
long	httpPayloadDeclaredLength content-length header, iff HTTP and present, null otherwise
long	recordDeclaredLength record length from metadata line
long	uncompressedBytes actual payload data (not including trailing newline), should match record-declared-length

Fields inherited from class org.archive.io.ArchiveRecord

digest, eor, header, in, MIN_HTTP_HEADER_LENGTH, position, strict

Fields inherited from interface org.archive.format.arc.ARCConstants

ACTUAL_LENGTH_KEY, ALEXA_DAT_MIME, ARC_FILE_EXTENSION, ARC_GZIP_EXTRA_FIELD, ARC_MAGIC_NUMBER, ARC_META_CHARSET, CARRIAGE_RETURN_ORD, CHECKSUM_FIELD_KEY, CHECKSUM_HEADER_FIELD_KEY, CODE_HEADER_FIELD_KEY, COMPRESSED_ARC_FILE_EXTENSION, DATE_STRING_KEY, DECLARED_LENGTH_KEY, DEFAULT_ENCODING, DEFAULT_GZIP_HEADER_LENGTH, DEFAULT_MAX_ARC_FILE_SIZE, DEFAULT_MIME, DELIMITER, DNS_MIME, DOT_ARC_FILE_EXTENSION, DOT_COMPRESSED_ARC_FILE_EXTENSION, DOT_COMPRESSED_FILE_EXTENSION, FIELD_COUNT, FILEDESC_SCHEME, FILENAME_FIELD_KEY, FILENAME_HEADER_FIELD_KEY, GZIP_HEADER_BEGIN, HEADER_FIELD_SEPARATOR, HEADER_LENGTH, IP_HEADER_FIELD_KEY, IP_KEY, LEADING_NEWLINES_KEY, LINE_SEPARATOR, LOCATION_HEADER_FIELD_KEY, MAX_HEADER_LINE_LENGTH, MAX_META_LENGTH, MAX_METADATA_LINE_LENGTH, MIME_KEY, MINIMUM_RECORD_LENGTH, NEW_LINE_ORD, OFFSET_FIELD_KEY, OFFSET_HEADER_FIELD_KEY, REQUIRED_VERSION_1_HEADER_FIELDS, STATUSCODE_FIELD_KEY, TOKENIZED_PREFIX, TRAILING_CRUFT_KEY, TRAILING_NEWLINES_KEY, URL_KEY

Fields inherited from interface org.archive.format.ArchiveFileConstants

ABSOLUTE_OFFSET_KEY, CDX, CDX_FILE, CDX_LINE_BUFFER_SIZE, CRLF, DATE_FIELD_KEY, DEFAULT_DIGEST_METHOD, DUMP, GZIP_DUMP, HEADER, INVALID_SUFFIX, LENGTH_FIELD_KEY, MIMETYPE_FIELD_KEY, NOHEAD, OCCUPIED_SUFFIX, ORIGIN_FIELD_KEY, READER_IDENTIFIER_FIELD_KEY, RECORD_IDENTIFIER_FIELD_KEY, SINGLE_SPACE, TYPE_FIELD_KEY, URL_FIELD_KEY, VERSION_FIELD_KEY

Constructor Summary

Constructors

Constructor and Description
ARCRecord(java.io.InputStream in, ArchiveRecordHeader metaData) Constructor.
ARCRecord(java.io.InputStream in, ArchiveRecordHeader metaData, int bodyOffset, boolean digest, boolean strict, boolean parseHttpHeaders) Constructor.
ARCRecord(java.io.InputStream in, java.lang.String identifier, long offset, boolean digest, boolean strict, boolean parseHttpHeaders) Constructor.
ARCRecord(java.io.InputStream in, java.lang.String identifier, long offset, boolean digest, boolean strict, boolean parseHttpHeaders, boolean isAlignedOnFirstRecord, java.lang.String version) Constructor.

Method Summary

Modifier and Type	Method and Description
void	dumpHttpHeader()
int	getBodyOffset()
protected java.lang.String	getDigest4Cdx(ArchiveRecordHeader h)
java.util.List<ARCConstants.ArcRecordErrors>	getErrors()
java.lang.String	getHeaderString()
org.apache.commons.httpclient.Header[]	getHttpHeaders()
protected java.lang.String	getIp4Cdx(ArchiveRecordHeader h)
ARCRecordMetaData	getMetaData()
int	getStatusCode() Return status code for this record.
protected java.lang.String	getStatusCode4Cdx(ArchiveRecordHeader h)
boolean	hasErrors()
int	read()
int	read(byte[] b, int offset, int length)
void	skipHttpHeader() Skip over the the http header if one present.

Methods inherited from class org.archive.io.ArchiveRecord

available, close, dump, dump, getDigestStr, getHeader, getIn, getMimetype4Cdx, getPosition, hasContentHeaders, incrementPosition, incrementPosition, isEor, isStrict, markSupported, outputCdx, setBodyOffset, setEor, setHeader, setStrict, skip, skip

Methods inherited from class java.io.InputStream

mark, read, reset

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

httpHeaderBytesRead

public long httpHeaderBytesRead

Http header bytes read while trying to read http header

recordDeclaredLength

public long recordDeclaredLength

record length from metadata line

compressedBytes

public long compressedBytes

null if source was not compressed

uncompressedBytes

public long uncompressedBytes

actual payload data (not including trailing newline), should match record-declared-length

httpPayloadDeclaredLength

public long httpPayloadDeclaredLength

content-length header, iff HTTP and present, null otherwise

httpPayloadActualLength

public long httpPayloadActualLength

actual http payload length, should match http-payload-declared-length

errors

public java.util.List<ARCConstants.ArcRecordErrors> errors

errors encountered reading record

Constructor Detail

ARCRecord

public ARCRecord(java.io.InputStream in,
                 ArchiveRecordHeader metaData)
          throws java.io.IOException

Constructor.

Parameters:

in - Stream cue'd up to be at the start of the record this instance is to represent.

metaData - Meta data.

Throws:

java.io.IOException

ARCRecord

public ARCRecord(java.io.InputStream in,
                 ArchiveRecordHeader metaData,
                 int bodyOffset,
                 boolean digest,
                 boolean strict,
                 boolean parseHttpHeaders)
          throws java.io.IOException

Constructor.

Parameters:

in - Stream cue'd up to be at the start of the record this instance is to represent.

metaData - Meta data.

bodyOffset - Offset into the body. Usually 0.

digest - True if we're to calculate digest for this record. Not digesting saves about ~15% of cpu during an ARC parse.

strict - Be strict parsing (Parsing stops if ARC inproperly formatted).

parseHttpHeaders - True if we are to parse HTTP headers. Costs about ~20% of CPU during an ARC parse.

Throws:

java.io.IOException

ARCRecord

public ARCRecord(java.io.InputStream in,
                 java.lang.String identifier,
                 long offset,
                 boolean digest,
                 boolean strict,
                 boolean parseHttpHeaders,
                 boolean isAlignedOnFirstRecord,
                 java.lang.String version)
          throws java.io.IOException

Constructor.

Parameters:

in - Stream cue'd up to be at the start of the records metadata this instance is to represent.

identifier - Identifier for this the hosting Reader.

offset - Current offset into in (Used to keep position properly aligned). Usually 0.

digest - True if we're to calculate digest for this record. Not digesting saves about ~15% of cpu during an ARC parse.

strict - Be strict parsing (Parsing stops if ARC inproperly formatted).

parseHttpHeaders - True if we are to parse HTTP headers. Costs about ~20% of CPU during an ARC parse.

isAllignedOnFirstRecord - True if this is the first record to be read from an archive

String - version Version information to be returned to the ARCReader constructing this record

Throws:

java.io.IOException

ARCRecord

public ARCRecord(java.io.InputStream in,
                 java.lang.String identifier,
                 long offset,
                 boolean digest,
                 boolean strict,
                 boolean parseHttpHeaders)
          throws java.io.IOException

Constructor.

Parameters:

in - Stream cue'd up to be at the start of the records metadata this instance is to represent.

identifier - Identifier for this the hosting Reader.

offset - Current offset into in (Used to keep position properly aligned). Usually 0.

digest - True if we're to calculate digest for this record. Not digesting saves about ~15% of cpu during an ARC parse.

strict - Be strict parsing (Parsing stops if ARC inproperly formatted).

parseHttpHeaders - True if we are to parse HTTP headers. Costs about ~20% of CPU during an ARC parse.

Throws:

java.io.IOException

Method Detail

getHeaderString

public java.lang.String getHeaderString()

skipHttpHeader

public void skipHttpHeader()
                    throws java.io.IOException

Skip over the the http header if one present. Subsequent reads will get the body.

Calling this method in the midst of reading the header will make for strange results. Otherwise, safe to call at any time though before reading any of the arc record content is only time that it makes sense.

After calling this method, you can call getHttpHeaders() to get the read http header.

Throws:

java.io.IOException

dumpHttpHeader

public void dumpHttpHeader()
                    throws java.io.IOException

Throws:

java.io.IOException

getStatusCode

public int getStatusCode()

Return status code for this record. This method will return -1 until the http header has been read.

Returns:

Status code.

getMetaData

public ARCRecordMetaData getMetaData()

Returns:

Meta data for this record.

getHttpHeaders

public org.apache.commons.httpclient.Header[] getHttpHeaders()

Returns:

http headers (Only available after header has been read).

getErrors

public java.util.List<ARCConstants.ArcRecordErrors> getErrors()

Returns:

ArcRecordErrors encountered when reading

hasErrors

public boolean hasErrors()

Returns:

true if ARC record errors found

read

public int read()
         throws java.io.IOException

Overrides:

read in class ArchiveRecord

Returns:

Next character in this ARCRecord's content else -1 if at end of this record.

Throws:

java.io.IOException

read

public int read(byte[] b,
                int offset,
                int length)
         throws java.io.IOException

Overrides:

read in class ArchiveRecord

Throws:

java.io.IOException

getBodyOffset

public int getBodyOffset()

Returns:

Offset at which the body begins (Only known after header has been read) or -1 if none or if we haven't read headers yet. Usually length of HTTP headers (does not include ARC metadata line length).

getIp4Cdx

protected java.lang.String getIp4Cdx(ArchiveRecordHeader h)

Overrides:

getIp4Cdx in class ArchiveRecord

getStatusCode4Cdx

protected java.lang.String getStatusCode4Cdx(ArchiveRecordHeader h)

Overrides:

getStatusCode4Cdx in class ArchiveRecord

getDigest4Cdx

protected java.lang.String getDigest4Cdx(ArchiveRecordHeader h)

Overrides:

getDigest4Cdx in class ArchiveRecord