Class ArchiveReader

java.lang.Object
org.archive.io.ArchiveReader
All Implemented Interfaces:
Closeable, AutoCloseable, Iterable<ArchiveRecord>
Direct Known Subclasses:
ARCReader, WARCReader
public abstract class ArchiveReader extends Object implements Iterable<ArchiveRecord>, Closeable
Reader for an Archive file of Archive ArchiveRecords.
Version:
$Date$ $Version$
Author:
stack

  • Field Details

    • in

      protected InputStream in
      Archive file input stream. Keep it around so we can close it when done.

      Set in constructor. Should support at least 1 byte mark/reset. Make it protected so subclasses have access.

    • MAX_ALLOWED_RECOVERABLES

      public static final int MAX_ALLOWED_RECOVERABLES
      Maximum amount of recoverable exceptions in a row. If more than this amount in a row, we'll let out the exception rather than go back in for yet another retry.
      See Also:

  • Constructor Details

    • ArchiveReader

      protected ArchiveReader()

  • Method Details

    • initialize

      protected void initialize(String i)
      Convenience method used by subclass constructors.
      Parameters:
      i - Identifier for Archive file this reader goes against.

    • getInputStream

      protected InputStream getInputStream(File f, long offset) throws IOException
      Convenience method for constructors.
      Parameters:
      f - File to read.
      offset - Offset at which to start reading.
      Returns:
      InputStream to read from.
      Throws:
      IOException - If failed open or fail to get a memory mapped byte buffer on file.

    • isCompressed

      public boolean isCompressed()

    • get

      public ArchiveRecord get(long offset) throws IOException
      Get record at passed offset.
      Parameters:
      offset - Byte index into file at which a record starts.
      Returns:
      An Archive Record reference.
      Throws:
      IOException

    • get

      public ArchiveRecord get() throws IOException
      Returns:
      Return Archive Record created against current offset.
      Throws:
      IOException

    • close

      public void close() throws IOException
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException

    • cleanupCurrentRecord

      protected void cleanupCurrentRecord() throws IOException
      Cleanout the current record if there is one.
      Throws:
      IOException

    • createArchiveRecord

      protected abstract ArchiveRecord createArchiveRecord(InputStream is, long offset) throws IOException
      Return an Archive Record homed on offset into is.
      Parameters:
      is - Stream to read Record from.
      offset - Offset to find Record at.
      Returns:
      ArchiveRecord instance.
      Throws:
      IOException

    • gotoEOR

      protected abstract void gotoEOR(ArchiveRecord record) throws IOException
      Skip over any trailing new lines at end of the record so we're lined up ready to read the next.
      Parameters:
      record -
      Throws:
      IOException

    • getFileExtension

      public abstract String getFileExtension()

    • getDotFileExtension

      public abstract String getDotFileExtension()

    • getVersion

      public String getVersion()
      Returns:
      Version of this Archive file.

    • validate

      public List<ArchiveRecordHeader> validate() throws IOException
      Validate the Archive file. This method iterates over the file throwing exception if it fails to successfully parse any record.

      Assumes the stream is at the start of the file.

      Returns:
      List of all read Archive Headers.
      Throws:
      IOException

    • validate

      public List<ArchiveRecordHeader> validate(int numRecords) throws IOException
      Validate the Archive file. This method iterates over the file throwing exception if it fails to successfully parse.

      We start validation from wherever we are in the stream.

      Parameters:
      numRecords - Number of records expected. Pass -1 if number is unknown.
      Returns:
      List of all read metadatas. As we validate records, we add a reference to the read metadata.
      Throws:
      IOException

    • isValid

      public boolean isValid()
      Test Archive file is valid. Assumes the stream is at the start of the file. Be aware that this method makes a pass over the whole file.
      Returns:
      True if file can be successfully parsed.

    • isStrict

      public boolean isStrict()
      Returns:
      Returns the strict.

    • setStrict

      public void setStrict(boolean s)
      Parameters:
      s - The strict to set.

    • setDigest

      public void setDigest(boolean d)
      Parameters:
      d - True if we're to digest.

    • isDigest

      public boolean isDigest()
      Returns:
      True if we're digesting as we read.

    • getLogger

      protected Logger getLogger()

    • iterator

      public Iterator<ArchiveRecord> iterator()
      Returns an ArchiveRecord iterator. Of note, on IOException, especially if ZipException reading compressed ARCs, rather than fail the iteration, try moving to the next record. If strict is not set, this will usually succeed.
      Specified by:
      iterator in interface Iterable<ArchiveRecord>
      Returns:
      An iterator over ARC records.

    • setCompressed

      protected void setCompressed(boolean compressed)

    • getCurrentRecord

      protected ArchiveRecord getCurrentRecord()
      Returns:
      The current ARC record or null if none. After construction has the arcfile header record.
      See Also:

    • currentRecord

      protected ArchiveRecord currentRecord(ArchiveRecord r)

    • getIn

      protected InputStream getIn()

    • setIn

      protected void setIn(InputStream in)

    • setVersion

      protected void setVersion(String version)

    • getReaderIdentifier

      public String getReaderIdentifier()

    • setReaderIdentifier

      protected void setReaderIdentifier(String i)

    • logStdErr

      public void logStdErr(Level level, String message)
      Log on stderr. Logging should go via the logging system. This method bypasses the logging system going direct to stderr. Should not generally be used. Its used for rare messages that come of cmdline usage of ARCReader ERRORs and WARNINGs. Override if using ARCReader in a context where no stderr or where you'd like to redirect stderr to other than System.err.
      Parameters:
      level - Level to log message at.
      message - Message to log.

    • positionForRecord

      protected static long positionForRecord(InputStream in)

    • stripExtension

      protected static String stripExtension(String name, String ext)

    • getFileName

      public String getFileName()
      Returns:
      short name of Archive file.

    • getStrippedFileName

      public String getStrippedFileName()
      Returns:
      short name of Archive file.

    • getStrippedFileName

      public static String getStrippedFileName(String name, String dotFileExtension)
      Parameters:
      name - Name of ARCFile.
      dotFileExtension - '.arc' or '.warc', etc.
      Returns:
      short name of Archive file.

    • getTrueOrFalse

      protected static boolean getTrueOrFalse(String value)
      Parameters:
      value - Value to test.
      Returns:
      True if value is 'true', else false.

    • output

      protected boolean output(String format) throws IOException, ParseException
      Parameters:
      format - Format to use outputting.
      Returns:
      True if handled.
      Throws:
      IOException
      ParseException

    • cdxOutput

      protected void cdxOutput(boolean toFile) throws IOException
      Throws:
      IOException

    • outputRecord

      public boolean outputRecord(String format) throws IOException
      Output passed record using passed format specifier.
      Parameters:
      format - What format to use outputting.
      Returns:
      True if handled.
      Throws:
      IOException

    • dump

      public abstract void dump(boolean compress) throws IOException, ParseException
      Dump this file on STDOUT
      Parameters:
      compress - True if dumped output is compressed.
      Throws:
      IOException
      ParseException

    • getDeleteFileOnCloseReader

      public abstract ArchiveReader getDeleteFileOnCloseReader(File f)
      Returns:
      an ArchiveReader that will delete a local file on close. Used when we bring Archive files local and need to clean up afterward.

    • outputRecord

      protected static void outputRecord(ArchiveReader r, String format) throws IOException
      Output passed record using passed format specifier.
      Parameters:
      r - ARCReader instance to output.
      format - What format to use outputting.
      Throws:
      IOException

    • getOptions

      protected static org.apache.commons.cli.Options getOptions()
      Returns:
      Base Options object filled out with help, digest, strict, etc. options.