Class ArchiveRecord

java.lang.Object
java.io.InputStream
org.archive.io.ArchiveRecord
All Implemented Interfaces:
Closeable, AutoCloseable
Direct Known Subclasses:
ARCRecord, HeaderedArchiveRecord, WARCRecord
public abstract class ArchiveRecord extends InputStream
Archive file Record.
Version:
$Date$ $Version$
Author:
stack

  • Field Details

    • MIN_HTTP_HEADER_LENGTH

      protected static final long MIN_HTTP_HEADER_LENGTH
      Minimal http response or request header length. I've seen in arcs content length of 1 with no header.

    • in

      protected InputStream in
      Stream to read this record from. Stream can only be read sequentially. Will only return this records' content returning a -1 if you try to read beyond the end of the current record.

      Streams can be markable or not. If they are, we'll be able to roll back when we've read too far. If not markable, assumption is that the underlying stream is managing our not reading too much (This pertains to the skipping over the end of the ARCRecord. See skip().

    • position

      protected long position
      Position w/i the Record content, within in. This position is relative within this Record. Its not same as the Archive file position.

    • eor

      protected boolean eor
      Set flag when we've reached the end-of-record.

    • digest

      protected MessageDigest digest
      Compute digest on what we read and add to metadata when done. Currently hardcoded as sha-1. TODO: Remove when archive records digest or else, add a facility that allows the arc reader to compare the calculated digest to that which is recorded in the arc.

      Protected instead of private so subclasses can update and complete the digest.

    • strict

      protected boolean strict

  • Constructor Details

    • ArchiveRecord

      public ArchiveRecord(InputStream in) throws IOException
      Constructor.
      Parameters:
      in - Stream cue'd up to be at the start of the record this instance is to represent.
      Throws:
      IOException

    • ArchiveRecord

      public ArchiveRecord(InputStream in, ArchiveRecordHeader header) throws IOException
      Constructor.
      Parameters:
      in - Stream cue'd up to be at the start of the record this instance is to represent.
      header - Header data.
      Throws:
      IOException

    • ArchiveRecord

      public ArchiveRecord(InputStream in, ArchiveRecordHeader header, int bodyOffset, boolean digest, boolean strict) throws IOException
      Constructor.
      Parameters:
      in - Stream cue'd up to be at the start of the record this instance is to represent.
      header - Header data.
      bodyOffset - Offset into the body. Usually 0.
      digest - True if we're to calculate digest for this record. Not digesting saves about ~15% of cpu during an ARC parse.
      strict - Be strict parsing (Parsing stops if ARC inproperly formatted).
      Throws:
      IOException

  • Method Details

    • markSupported

      public boolean markSupported()
      Overrides:
      markSupported in class InputStream

    • getHeader

      public ArchiveRecordHeader getHeader()
      Returns:
      Header data for this record.

    • setHeader

      protected void setHeader(ArchiveRecordHeader header)

    • close

      public void close() throws IOException
      Calling close on a record skips us past this record to the next record in the stream. It does not actually close the stream. The underlying steam is probably being used by the next arc record.
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Overrides:
      close in class InputStream
      Throws:
      IOException

    • read

      public int read() throws IOException
      Specified by:
      read in class InputStream
      Returns:
      Next character in this Record content else -1 if at EOR.
      Throws:
      IOException

    • read

      public int read(byte[] b, int offset, int length) throws IOException
      Overrides:
      read in class InputStream
      Throws:
      IOException

    • available

      public int available()
      This available is not the stream's available. Its an available based on what the stated Archive record length is minus what we've read to date.
      Overrides:
      available in class InputStream
      Returns:
      True if bytes remaining in record content.

    • skip

      protected void skip() throws IOException
      Skip over this records content.
      Throws:
      IOException

    • skip

      public long skip(long n) throws IOException
      Overrides:
      skip in class InputStream
      Throws:
      IOException

    • isStrict

      public boolean isStrict()
      Returns:
      Returns the strict.

    • setStrict

      public void setStrict(boolean strict)
      Parameters:
      strict - The strict to set.

    • getIn

      protected InputStream getIn()

    • getDigestStr

      public String getDigestStr()

    • incrementPosition

      protected void incrementPosition()

    • incrementPosition

      protected void incrementPosition(long incr)

    • getPosition

      public long getPosition()

    • isEor

      protected boolean isEor()

    • setEor

      protected void setEor(boolean eor)

    • getStatusCode4Cdx

      protected String getStatusCode4Cdx(ArchiveRecordHeader h)

    • getIp4Cdx

      protected String getIp4Cdx(ArchiveRecordHeader h)

    • getDigest4Cdx

      protected String getDigest4Cdx(ArchiveRecordHeader h)

    • getMimetype4Cdx

      protected String getMimetype4Cdx(ArchiveRecordHeader h)

    • outputCdx

      protected String outputCdx(String strippedFileName) throws IOException
      Throws:
      IOException

    • dump

      public void dump() throws IOException
      Writes output on STDOUT.
      Throws:
      IOException

    • dump

      public void dump(OutputStream os) throws IOException
      Writes output on passed os.
      Throws:
      IOException

    • hasContentHeaders

      public boolean hasContentHeaders()
      Is it likely that this record contains headers? This method will return true if the body is a http response that includes http response headers or the body is a http request that includes request headers, etc. Be aware that headers in content are distinct from ArchiveRecordHeader 'headers'.
      Returns:
      True if this Record's content has headers:

    • setBodyOffset

      protected void setBodyOffset(int bodyOffset)