Class ARCReader

java.lang.Object
org.archive.io.ArchiveReader
org.archive.io.arc.ARCReader
All Implemented Interfaces:
Closeable, AutoCloseable, Iterable<ArchiveRecord>
Direct Known Subclasses:
ARCReaderFactory.CompressedARCReader, ARCReaderFactory.UncompressedARCReader
public abstract class ARCReader extends ArchiveReader implements Closeable
Get an iterator on an ARC file or get a record by absolute position. ARC files are described here: Arc File Format.

This class knows how to parse an ARC file. Pass it a file path or an URL to an ARC. It can parse ARC Version 1 and 2.

Iterator returns ARCRecord though Iterator.next() is returning java.lang.Object. Cast the return.

Profiling java.io vs. memory-mapped ByteBufferInputStream shows the latter slightly slower -- but not by much. TODO: Test more. Just change ArchiveReader.getInputStream(File, long).

Version:
$Date$ $Revision$
Author:
stack

  • Constructor Details

    • ARCReader

      protected ARCReader()

  • Method Details

    • gotoEOR

      protected void gotoEOR(ArchiveRecord record) throws IOException
      Skip over any trailing new lines at end of the record so we're lined up ready to read the next.
      Specified by:
      gotoEOR in class ArchiveReader
      Parameters:
      record -
      Throws:
      IOException

    • createArchiveRecord

      protected ARCRecord createArchiveRecord(InputStream is, long offset) throws IOException
      Create new arc record. Encapsulate housekeeping that has to do w/ creating a new record.

      Call this method at end of constructor to read in the arcfile header. Will be problems reading subsequent arc records if you don't since arcfile header has the list of metadata fields for all records that follow.

      When parsing through ARCs writing out CDX info, we spend about 38% of CPU in here -- about 30% of which is in getTokenizedHeaderLine -- of which 16% is reading.

      Specified by:
      createArchiveRecord in class ArchiveReader
      Parameters:
      is - InputStream to use.
      offset - Absolute offset into arc file.
      Returns:
      An arc record.
      Throws:
      IOException

    • getVersion

      public String getVersion()
      Returns version of this ARC file. Usually read from first record of ARC. If we're reading without having first read the first record -- e.g. random access into middle of an ARC -- then version will not have been set. For now, we return a default, version 1.1. Later, if more than just one version of ARC, we could look at such as the meta line to see what version of ARC this is.
      Overrides:
      getVersion in class ArchiveReader
      Returns:
      Version of this ARC file.

    • isAlignedOnFirstRecord

      protected boolean isAlignedOnFirstRecord()

    • setAlignedOnFirstRecord

      protected void setAlignedOnFirstRecord(boolean alignedOnFirstRecord)

    • isParseHttpHeaders

      public boolean isParseHttpHeaders()
      Returns:
      Returns the parseHttpHeaders.

    • setParseHttpHeaders

      public void setParseHttpHeaders(boolean parse)
      Parameters:
      parse - The parseHttpHeaders to set.

    • getFileExtension

      public String getFileExtension()
      Specified by:
      getFileExtension in class ArchiveReader

    • getDotFileExtension

      public String getDotFileExtension()
      Specified by:
      getDotFileExtension in class ArchiveReader

    • output

      protected boolean output(String format) throws IOException, ParseException
      Overrides:
      output in class ArchiveReader
      Parameters:
      format - Format to use outputting.
      Returns:
      True if handled.
      Throws:
      IOException
      ParseException

    • outputRecord

      public boolean outputRecord(String format) throws IOException
      Description copied from class: ArchiveReader
      Output passed record using passed format specifier.
      Overrides:
      outputRecord in class ArchiveReader
      Parameters:
      format - What format to use outputting.
      Returns:
      True if handled.
      Throws:
      IOException

    • dump

      public void dump(boolean compress) throws IOException, ParseException
      Description copied from class: ArchiveReader
      Dump this file on STDOUT
      Specified by:
      dump in class ArchiveReader
      Parameters:
      compress - True if dumped output is compressed.
      Throws:
      IOException
      ParseException

    • getDeleteFileOnCloseReader

      public ARCReader getDeleteFileOnCloseReader(File f)
      Specified by:
      getDeleteFileOnCloseReader in class ArchiveReader
      Returns:
      an ArchiveReader that will delete a local file on close. Used when we bring Archive files local and need to clean up afterward.

    • output

      protected static void output(ARCReader reader, String format) throws IOException, ParseException
      Write out the arcfile.
      Parameters:
      reader -
      format - Format to use outputting.
      Throws:
      IOException
      ParseException

    • createCDXIndexFile

      public static void createCDXIndexFile(String urlOrPath) throws IOException, ParseException
      Generate a CDX index file for an ARC file.
      Parameters:
      urlOrPath - The ARC file to generate a CDX index for
      Throws:
      IOException
      ParseException

    • main

      public static void main(String[] args) throws org.apache.commons.cli.ParseException, IOException, ParseException
      Command-line interface to ARCReader. Here is the command-line interface: 
       usage: java org.archive.io.arc.ARCReader [--offset=#] ARCFILE
  -h,--help      Prints this message and exits.
  -o,--offset    Outputs record at this offset into arc file.

      See in $HERITRIX_HOME/bin/arcreader for a script that'll take care of classpaths and the calling of ARCReader.

      Outputs using a pseudo-CDX format as described here: CDX Legent and here Example. Legend used in below is: 'CDX b e a m s c V (or v if uncompressed) n g'. Hash is hard-coded straight SHA-1 hash of content.

      Parameters:
      args - Command-line arguments.
      Throws:
      org.apache.commons.cli.ParseException - Failed parse of the command line.
      IOException
      ParseException