Class WARCRecord

java.lang.Object

java.io.InputStream

org.archive.io.ArchiveRecord

org.archive.io.warc.WARCRecord

All Implemented Interfaces:

Closeable, AutoCloseable

public class WARCRecord
extends ArchiveRecord

A WARC file Record.

Author:

stack

Field Summary

Fields inherited from class org.archive.io.ArchiveRecord

digest, eor, header, in, MIN_HTTP_HEADER_LENGTH, position, strict

Constructor Summary

Constructors

Constructor	Description
WARCRecord(InputStream in, String identifier, long offset)	Constructor.
WARCRecord(InputStream in, String identifier, long offset, boolean digest, boolean strict)	Constructor.
WARCRecord(InputStream in, ArchiveRecordHeader headers)	Constructor.

Method Summary

Modifier and Type	Method	Description
protected String	getMimetype4Cdx(ArchiveRecordHeader h)
protected ArchiveRecordHeader	parseHeaders(InputStream in, String identifier, long offset, boolean strict)	Parse WARC Header Line and Named Fields.

Methods inherited from class org.archive.io.ArchiveRecord

available, close, dump, dump, getDigest4Cdx, getDigestStr, getHeader, getIn, getIp4Cdx, getPosition, getStatusCode4Cdx, hasContentHeaders, incrementPosition, incrementPosition, isEor, isStrict, markSupported, outputCdx, read, read, setBodyOffset, setEor, setHeader, setStrict, skip, skip

Methods inherited from class java.io.InputStream

mark, read, reset

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

WARCRecord

public WARCRecord(InputStream in,
 String identifier,
 long offset)
           throws IOException

Constructor.

Parameters:

in - Stream cue'd up to be at the start of the record this instance is to represent.

Throws:

IOException

WARCRecord

public WARCRecord(InputStream in,
 ArchiveRecordHeader headers)
           throws IOException

Constructor.

Parameters:

in - Stream cue'd up just past Header Line and Named Fields.

headers - Header Line and ANVL Named fields.

Throws:

IOException

WARCRecord

public WARCRecord(InputStream in,
 String identifier,
 long offset,
 boolean digest,
 boolean strict)
           throws IOException

Constructor.

Parameters:

in - Stream cue'd up to be at the start of the record this instance is to represent or, if headers is not null, just past the Header Line and Named Fields.

identifier - Identifier for this the hosting Reader.

offset - Current offset into in (Used to keep position properly aligned). Usually 0.

digest - True if we're to calculate digest for this record. Not digesting saves about ~15% of cpu during parse.

strict - Be strict parsing (Parsing stops if file inproperly formatted).

Throws:

IOException

Method Details

parseHeaders

protected ArchiveRecordHeader parseHeaders(InputStream in,
 String identifier,
 long offset,
 boolean strict)
                                    throws IOException

Parse WARC Header Line and Named Fields.

Parameters:

in - Stream to read.

identifier - Identifier for the hosting Reader.

offset - Absolute offset into Reader.

strict - Whether to be loose parsing or not.

Returns:

An ArchiveRecordHeader.

Throws:

IOException

getMimetype4Cdx

protected String getMimetype4Cdx(ArchiveRecordHeader h)

Overrides:

getMimetype4Cdx in class ArchiveRecord