org.archive.io.warc

Class WARCReader

java.lang.Object

org.archive.io.ArchiveReader

org.archive.io.warc.WARCReader

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, java.lang.Iterable<ArchiveRecord>, ArchiveFileConstants, WARCConstants

Direct Known Subclasses:

WARCReaderFactory.CompressedWARCReader, WARCReaderFactory.UncompressedWARCReader

public class WARCReader
extends ArchiveReader
implements WARCConstants

WARCReader. Go via WARCReaderFactory to get instance.

Version:

$Date: 2006-11-27 18:03:03 -0800 (Mon, 27 Nov 2006) $ $Version$

Author:

stack

Nested Class Summary

Nested classes/interfaces inherited from class org.archive.io.ArchiveReader

ArchiveReader.ArchiveRecordIterator

Nested classes/interfaces inherited from interface org.archive.format.warc.WARCConstants

WARCConstants.WARCRecordType

Field Summary

Fields inherited from class org.archive.io.ArchiveReader

in, MAX_ALLOWED_RECOVERABLES

Fields inherited from interface org.archive.format.warc.WARCConstants

COLON_SPACE, COMPRESSED_WARC_FILE_EXTENSION, CONTENT_DESCRIPTION, CONTENT_LENGTH, CONTENT_TYPE, DEFAULT_ENCODING, DEFAULT_MAX_WARC_FILE_SIZE, DOT_COMPRESSED_WARC_FILE_EXTENSION, DOT_WARC_FILE_EXTENSION, FTP_CONTROL_CONVERSATION_MIMETYPE, HEADER_FIELD_SEPARATOR, HEADER_KEY_BLOCK_DIGEST, HEADER_KEY_CONCURRENT_TO, HEADER_KEY_DATE, HEADER_KEY_ETAG, HEADER_KEY_FILENAME, HEADER_KEY_ID, HEADER_KEY_IP, HEADER_KEY_LAST_MODIFIED, HEADER_KEY_PAYLOAD_DIGEST, HEADER_KEY_PROFILE, HEADER_KEY_REFERS_TO, HEADER_KEY_REFERS_TO_DATE, HEADER_KEY_REFERS_TO_FILE_OFFSET, HEADER_KEY_REFERS_TO_FILENAME, HEADER_KEY_REFERS_TO_TARGET_URI, HEADER_KEY_TRUNCATED, HEADER_KEY_TYPE, HEADER_KEY_URI, HEADER_LINE_ENCODING, HTTP_REQUEST_MIMETYPE, HTTP_RESPONSE_MIMETYPE, HTTP_RESPONSE_MIMETYPE_NS, MAX_LINE_LENGTH, MAX_WARC_HEADER_LINE_LENGTH, NAMED_FIELD_CHECKSUM_LABEL, NAMED_FIELD_DESCRIPTION, NAMED_FIELD_FILEDESC, NAMED_FIELD_IP_LABEL, NAMED_FIELD_RELATED_LABEL, NAMED_FIELD_TRUNCATED, NAMED_FIELD_TRUNCATED_VALUE_HEAD, NAMED_FIELD_TRUNCATED_VALUE_LENGTH, NAMED_FIELD_TRUNCATED_VALUE_TIME, NAMED_FIELD_TRUNCATED_VALUE_UNSPECIFIED, NAMED_FIELD_WARCFILENAME, PLACEHOLDER_RECORD_LENGTH_STRING, PROFILE_REVISIT_IDENTICAL_DIGEST, PROFILE_REVISIT_NOT_MODIFIED, TRUNCATED_VALUE_UNSPECIFIED, TYPE, WARC_FIELDS_TYPE, WARC_FILE_EXTENSION, WARC_HEADER_ENCODING, WARC_ID, WARC_MAGIC, WARC_VERSION, WSP

Fields inherited from interface org.archive.format.ArchiveFileConstants

ABSOLUTE_OFFSET_KEY, CDX, CDX_FILE, CDX_LINE_BUFFER_SIZE, CRLF, DATE_FIELD_KEY, DEFAULT_DIGEST_METHOD, DOT_COMPRESSED_FILE_EXTENSION, DUMP, GZIP_DUMP, HEADER, INVALID_SUFFIX, LENGTH_FIELD_KEY, MIMETYPE_FIELD_KEY, NOHEAD, OCCUPIED_SUFFIX, ORIGIN_FIELD_KEY, READER_IDENTIFIER_FIELD_KEY, RECORD_IDENTIFIER_FIELD_KEY, SINGLE_SPACE, TYPE_FIELD_KEY, URL_FIELD_KEY, VERSION_FIELD_KEY

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	WARCReader()

Method Summary

Modifier and Type	Method and Description
protected WARCRecord	createArchiveRecord(java.io.InputStream is, long offset) Create new WARC record.
static void	createCDXIndexFile(java.lang.String urlOrPath) Generate a CDX index file for an ARC file.
void	dump(boolean compress) Dump this file on STDOUT
ArchiveReader	getDeleteFileOnCloseReader(java.io.File f)
java.lang.String	getDotFileExtension()
java.lang.String	getFileExtension()
protected void	gotoEOR(ArchiveRecord record) Skip over any trailing new lines at end of the record so we're lined up ready to read the next.
protected void	initialize(java.lang.String i) Convenience method used by subclass constructors.
static void	main(java.lang.String[] args) Command-line interface to WARCReader.
protected static void	output(WARCReader reader, java.lang.String format) Write out the arcfile.
protected void	readExpectedChar(java.io.InputStream is, int expected)

Methods inherited from class org.archive.io.ArchiveReader

cdxOutput, cleanupCurrentRecord, close, currentRecord, get, get, getCurrentRecord, getFileName, getIn, getInputStream, getLogger, getOptions, getReaderIdentifier, getStrippedFileName, getStrippedFileName, getTrueOrFalse, getVersion, isCompressed, isDigest, isStrict, isValid, iterator, logStdErr, output, outputRecord, outputRecord, positionForRecord, setCompressed, setDigest, setIn, setReaderIdentifier, setStrict, setVersion, stripExtension, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Constructor Detail

WARCReader

protected WARCReader()

Method Detail

initialize

protected void initialize(java.lang.String i)

Description copied from class: ArchiveReader

Convenience method used by subclass constructors.

Overrides:

initialize in class ArchiveReader

Parameters:

i - Identifier for Archive file this reader goes against.

gotoEOR

protected void gotoEOR(ArchiveRecord record)
                throws java.io.IOException

Skip over any trailing new lines at end of the record so we're lined up ready to read the next.

Specified by:

gotoEOR in class ArchiveReader

Parameters:

record -

Throws:

java.io.IOException

readExpectedChar

protected void readExpectedChar(java.io.InputStream is,
                                int expected)
                         throws java.io.IOException

Throws:

java.io.IOException

createArchiveRecord

protected WARCRecord createArchiveRecord(java.io.InputStream is,
                                         long offset)
                                  throws java.io.IOException

Create new WARC record. Encapsulate housekeeping that has to do w/ creating new Record.

Specified by:

createArchiveRecord in class ArchiveReader

Parameters:

is - InputStream to use.

offset - Absolute offset into WARC file.

Returns:

A WARCRecord.

Throws:

java.io.IOException

dump

public void dump(boolean compress)
          throws java.io.IOException,
                 java.text.ParseException

Description copied from class: ArchiveReader

Dump this file on STDOUT

Specified by:

dump in class ArchiveReader

Throws:

java.io.IOException

java.text.ParseException

getDeleteFileOnCloseReader

public ArchiveReader getDeleteFileOnCloseReader(java.io.File f)

Specified by:

getDeleteFileOnCloseReader in class ArchiveReader

Returns:

an ArchiveReader that will delete a local file on close. Used when we bring Archive files local and need to clean up afterward.

getDotFileExtension

public java.lang.String getDotFileExtension()

Specified by:

getDotFileExtension in class ArchiveReader

getFileExtension

public java.lang.String getFileExtension()

Specified by:

getFileExtension in class ArchiveReader

output

protected static void output(WARCReader reader,
                             java.lang.String format)
                      throws java.io.IOException,
                             java.text.ParseException

Write out the arcfile.

Parameters:

reader -

format - Format to use outputting.

Throws:

java.io.IOException

java.text.ParseException

createCDXIndexFile

public static void createCDXIndexFile(java.lang.String urlOrPath)
                               throws java.io.IOException,
                                      java.text.ParseException

Generate a CDX index file for an ARC file.

Parameters:

urlOrPath - The ARC file to generate a CDX index for

Throws:

java.io.IOException

java.text.ParseException

main

public static void main(java.lang.String[] args)
                 throws org.apache.commons.cli.ParseException,
                        java.io.IOException,
                        java.text.ParseException

Command-line interface to WARCReader. Here is the command-line interface:

 usage: java org.archive.io.arc.WARCReader [--offset=#] ARCFILE
  -h,--help      Prints this message and exits.
  -o,--offset    Outputs record at this offset into arc file.

Outputs using a pseudo-CDX format as described here: CDX Legent and here Example. Legend used in below is: 'CDX b e a m s c V (or v if uncompressed) n g'. Hash is hard-coded straight SHA-1 hash of content.

Parameters:

args - Command-line arguments.

Throws:

org.apache.commons.cli.ParseException - Failed parse of the command line.

java.io.IOException

java.text.ParseException