org.archive.io.arc

Class ARCReader

java.lang.Object

org.archive.io.ArchiveReader

org.archive.io.arc.ARCReader

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, java.lang.Iterable<ArchiveRecord>, ARCConstants, ArchiveFileConstants

Direct Known Subclasses:

ARCReaderFactory.CompressedARCReader, ARCReaderFactory.UncompressedARCReader

public abstract class ARCReader
extends ArchiveReader
implements ARCConstants, java.io.Closeable

Get an iterator on an ARC file or get a record by absolute position. ARC files are described here: Arc File Format.

This class knows how to parse an ARC file. Pass it a file path or an URL to an ARC. It can parse ARC Version 1 and 2.

Iterator returns ARCRecord though Iterator.next() is returning java.lang.Object. Cast the return.

Profiling java.io vs. memory-mapped ByteBufferInputStream shows the latter slightly slower -- but not by much. TODO: Test more. Just change ArchiveReader.getInputStream(File, long).

Version:

$Date$ $Revision$

Author:

stack

Nested Class Summary

Nested classes/interfaces inherited from class org.archive.io.ArchiveReader

ArchiveReader.ArchiveRecordIterator

Nested classes/interfaces inherited from interface org.archive.format.arc.ARCConstants

ARCConstants.ArcRecordErrors

Field Summary

Fields inherited from class org.archive.io.ArchiveReader

in, MAX_ALLOWED_RECOVERABLES

Fields inherited from interface org.archive.format.arc.ARCConstants

ACTUAL_LENGTH_KEY, ALEXA_DAT_MIME, ARC_FILE_EXTENSION, ARC_GZIP_EXTRA_FIELD, ARC_MAGIC_NUMBER, ARC_META_CHARSET, CARRIAGE_RETURN_ORD, CHECKSUM_FIELD_KEY, CHECKSUM_HEADER_FIELD_KEY, CODE_HEADER_FIELD_KEY, COMPRESSED_ARC_FILE_EXTENSION, DATE_STRING_KEY, DECLARED_LENGTH_KEY, DEFAULT_ENCODING, DEFAULT_GZIP_HEADER_LENGTH, DEFAULT_MAX_ARC_FILE_SIZE, DEFAULT_MIME, DELIMITER, DNS_MIME, DOT_ARC_FILE_EXTENSION, DOT_COMPRESSED_ARC_FILE_EXTENSION, DOT_COMPRESSED_FILE_EXTENSION, FIELD_COUNT, FILEDESC_SCHEME, FILENAME_FIELD_KEY, FILENAME_HEADER_FIELD_KEY, GZIP_HEADER_BEGIN, HEADER_FIELD_SEPARATOR, HEADER_LENGTH, IP_HEADER_FIELD_KEY, IP_KEY, LEADING_NEWLINES_KEY, LINE_SEPARATOR, LOCATION_HEADER_FIELD_KEY, MAX_HEADER_LINE_LENGTH, MAX_META_LENGTH, MAX_METADATA_LINE_LENGTH, MIME_KEY, MINIMUM_RECORD_LENGTH, NEW_LINE_ORD, OFFSET_FIELD_KEY, OFFSET_HEADER_FIELD_KEY, REQUIRED_VERSION_1_HEADER_FIELDS, STATUSCODE_FIELD_KEY, TOKENIZED_PREFIX, TRAILING_CRUFT_KEY, TRAILING_NEWLINES_KEY, URL_KEY

Fields inherited from interface org.archive.format.ArchiveFileConstants

ABSOLUTE_OFFSET_KEY, CDX, CDX_FILE, CDX_LINE_BUFFER_SIZE, CRLF, DATE_FIELD_KEY, DEFAULT_DIGEST_METHOD, DUMP, GZIP_DUMP, HEADER, INVALID_SUFFIX, LENGTH_FIELD_KEY, MIMETYPE_FIELD_KEY, NOHEAD, OCCUPIED_SUFFIX, ORIGIN_FIELD_KEY, READER_IDENTIFIER_FIELD_KEY, RECORD_IDENTIFIER_FIELD_KEY, SINGLE_SPACE, TYPE_FIELD_KEY, URL_FIELD_KEY, VERSION_FIELD_KEY

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	ARCReader()

Method Summary

Modifier and Type	Method and Description
protected ARCRecord	createArchiveRecord(java.io.InputStream is, long offset) Create new arc record.
static void	createCDXIndexFile(java.lang.String urlOrPath) Generate a CDX index file for an ARC file.
void	dump(boolean compress) Dump this file on STDOUT
ARCReader	getDeleteFileOnCloseReader(java.io.File f)
java.lang.String	getDotFileExtension()
java.lang.String	getFileExtension()
java.lang.String	getVersion() Returns version of this ARC file.
protected void	gotoEOR(ArchiveRecord record) Skip over any trailing new lines at end of the record so we're lined up ready to read the next.
protected boolean	isAlignedOnFirstRecord()
boolean	isParseHttpHeaders()
static void	main(java.lang.String[] args) Command-line interface to ARCReader.
protected static void	output(ARCReader reader, java.lang.String format) Write out the arcfile.
protected boolean	output(java.lang.String format)
boolean	outputRecord(java.lang.String format) Output passed record using passed format specifier.
protected void	setAlignedOnFirstRecord(boolean alignedOnFirstRecord)
void	setParseHttpHeaders(boolean parse)

Methods inherited from class org.archive.io.ArchiveReader

cdxOutput, cleanupCurrentRecord, close, currentRecord, get, get, getCurrentRecord, getFileName, getIn, getInputStream, getLogger, getOptions, getReaderIdentifier, getStrippedFileName, getStrippedFileName, getTrueOrFalse, initialize, isCompressed, isDigest, isStrict, isValid, iterator, logStdErr, outputRecord, positionForRecord, setCompressed, setDigest, setIn, setReaderIdentifier, setStrict, setVersion, stripExtension, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.io.Closeable

close

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Constructor Detail

ARCReader

protected ARCReader()

Method Detail

gotoEOR

protected void gotoEOR(ArchiveRecord record)
                throws java.io.IOException

Skip over any trailing new lines at end of the record so we're lined up ready to read the next.

Specified by:

gotoEOR in class ArchiveReader

Parameters:

record -

Throws:

java.io.IOException

createArchiveRecord

protected ARCRecord createArchiveRecord(java.io.InputStream is,
                                        long offset)
                                 throws java.io.IOException

Create new arc record. Encapsulate housekeeping that has to do w/ creating a new record.

Call this method at end of constructor to read in the arcfile header. Will be problems reading subsequent arc records if you don't since arcfile header has the list of metadata fields for all records that follow.

When parsing through ARCs writing out CDX info, we spend about 38% of CPU in here -- about 30% of which is in getTokenizedHeaderLine -- of which 16% is reading.

Specified by:

createArchiveRecord in class ArchiveReader

Parameters:

is - InputStream to use.

offset - Absolute offset into arc file.

Returns:

An arc record.

Throws:

java.io.IOException

getVersion

public java.lang.String getVersion()

Returns version of this ARC file. Usually read from first record of ARC. If we're reading without having first read the first record -- e.g. random access into middle of an ARC -- then version will not have been set. For now, we return a default, version 1.1. Later, if more than just one version of ARC, we could look at such as the meta line to see what version of ARC this is.

Overrides:

getVersion in class ArchiveReader

Returns:

Version of this ARC file.

isAlignedOnFirstRecord

protected boolean isAlignedOnFirstRecord()

setAlignedOnFirstRecord

protected void setAlignedOnFirstRecord(boolean alignedOnFirstRecord)

isParseHttpHeaders

public boolean isParseHttpHeaders()

Returns:

Returns the parseHttpHeaders.

setParseHttpHeaders

public void setParseHttpHeaders(boolean parse)

Parameters:

parse - The parseHttpHeaders to set.

getFileExtension

public java.lang.String getFileExtension()

Specified by:

getFileExtension in class ArchiveReader

getDotFileExtension

public java.lang.String getDotFileExtension()

Specified by:

getDotFileExtension in class ArchiveReader

output

protected boolean output(java.lang.String format)
                  throws java.io.IOException,
                         java.text.ParseException

Overrides:

output in class ArchiveReader

Parameters:

format - Format to use outputting.

Returns:

True if handled.

Throws:

java.io.IOException

java.text.ParseException

outputRecord

public boolean outputRecord(java.lang.String format)
                     throws java.io.IOException

Description copied from class: ArchiveReader

Output passed record using passed format specifier.

Overrides:

outputRecord in class ArchiveReader

Parameters:

format - What format to use outputting.

Returns:

True if handled.

Throws:

java.io.IOException

dump

public void dump(boolean compress)
          throws java.io.IOException,
                 java.text.ParseException

Description copied from class: ArchiveReader

Dump this file on STDOUT

Specified by:

dump in class ArchiveReader

Throws:

java.io.IOException

java.text.ParseException

getDeleteFileOnCloseReader

public ARCReader getDeleteFileOnCloseReader(java.io.File f)

Specified by:

getDeleteFileOnCloseReader in class ArchiveReader

Returns:

an ArchiveReader that will delete a local file on close. Used when we bring Archive files local and need to clean up afterward.

output

protected static void output(ARCReader reader,
                             java.lang.String format)
                      throws java.io.IOException,
                             java.text.ParseException

Write out the arcfile.

Parameters:

reader -

format - Format to use outputting.

Throws:

java.io.IOException

java.text.ParseException

createCDXIndexFile

public static void createCDXIndexFile(java.lang.String urlOrPath)
                               throws java.io.IOException,
                                      java.text.ParseException

Generate a CDX index file for an ARC file.

Parameters:

urlOrPath - The ARC file to generate a CDX index for

Throws:

java.io.IOException

java.text.ParseException

main

public static void main(java.lang.String[] args)
                 throws org.apache.commons.cli.ParseException,
                        java.io.IOException,
                        java.text.ParseException

Command-line interface to ARCReader. Here is the command-line interface:

 usage: java org.archive.io.arc.ARCReader [--offset=#] ARCFILE
  -h,--help      Prints this message and exits.
  -o,--offset    Outputs record at this offset into arc file.

See in $HERITRIX_HOME/bin/arcreader for a script that'll take care of classpaths and the calling of ARCReader.

Outputs using a pseudo-CDX format as described here: CDX Legent and here Example. Legend used in below is: 'CDX b e a m s c V (or v if uncompressed) n g'. Hash is hard-coded straight SHA-1 hash of content.

Parameters:

args - Command-line arguments.

Throws:

org.apache.commons.cli.ParseException - Failed parse of the command line.

java.io.IOException

java.text.ParseException