Class WARCReader

java.lang.Object

org.archive.io.ArchiveReader

org.archive.io.warc.WARCReader

All Implemented Interfaces:

Closeable, AutoCloseable, Iterable<ArchiveRecord>

Direct Known Subclasses:

WARCReaderFactory.CompressedWARCReader, WARCReaderFactory.UncompressedWARCReader

public class WARCReader
extends ArchiveReader

WARCReader. Go via WARCReaderFactory to get instance.

Version:

$Date: 2006-11-27 18:03:03 -0800 (Mon, 27 Nov 2006) $ $Version$

Author:

stack

Nested Class Summary

Nested classes/interfaces inherited from class org.archive.io.ArchiveReader

ArchiveReader.ArchiveRecordIterator

Field Summary

Fields inherited from class org.archive.io.ArchiveReader

in, MAX_ALLOWED_RECOVERABLES

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	WARCReader()

Method Summary

Modifier and Type	Method	Description
protected WARCRecord	createArchiveRecord(InputStream is, long offset)	Create new WARC record.
static void	createCDXIndexFile(String urlOrPath)	Generate a CDX index file for an ARC file.
void	dump(boolean compress)	Dump this file on STDOUT
ArchiveReader	getDeleteFileOnCloseReader(File f)
String	getDotFileExtension()
String	getFileExtension()
protected void	gotoEOR(ArchiveRecord record)	Skip over any trailing new lines at end of the record so we're lined up ready to read the next.
protected void	initialize(String i)	Convenience method used by subclass constructors.
static void	main(String[] args)	Command-line interface to WARCReader.
protected static void	output(WARCReader reader, String format)	Write out the arcfile.
protected void	readExpectedChar(InputStream is, int expected)

Methods inherited from class org.archive.io.ArchiveReader

cdxOutput, cleanupCurrentRecord, close, currentRecord, get, get, getCurrentRecord, getFileName, getIn, getInputStream, getLogger, getOptions, getReaderIdentifier, getStrippedFileName, getStrippedFileName, getTrueOrFalse, getVersion, isCompressed, isDigest, isStrict, isValid, iterator, logStdErr, output, outputRecord, outputRecord, positionForRecord, setCompressed, setDigest, setIn, setReaderIdentifier, setStrict, setVersion, stripExtension, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Constructor Details

WARCReader

protected WARCReader()

Method Details

initialize

protected void initialize(String i)

Description copied from class: ArchiveReader

Convenience method used by subclass constructors.

Overrides:

initialize in class ArchiveReader

Parameters:

i - Identifier for Archive file this reader goes against.

gotoEOR

protected void gotoEOR(ArchiveRecord record)
                throws IOException

Skip over any trailing new lines at end of the record so we're lined up ready to read the next.

Specified by:

gotoEOR in class ArchiveReader

Parameters:

record -

Throws:

IOException

readExpectedChar

protected void readExpectedChar(InputStream is,
 int expected)
                         throws IOException

Throws:

IOException

createArchiveRecord

protected WARCRecord createArchiveRecord(InputStream is,
 long offset)
                                  throws IOException

Create new WARC record. Encapsulate housekeeping that has to do w/ creating new Record.

Specified by:

createArchiveRecord in class ArchiveReader

Parameters:

is - InputStream to use.

offset - Absolute offset into WARC file.

Returns:

A WARCRecord.

Throws:

IOException

dump

public void dump(boolean compress)
          throws IOException,
ParseException

Description copied from class: ArchiveReader

Dump this file on STDOUT

Specified by:

dump in class ArchiveReader

Parameters:

compress - True if dumped output is compressed.

Throws:

IOException

ParseException

getDeleteFileOnCloseReader

public ArchiveReader getDeleteFileOnCloseReader(File f)

Specified by:

getDeleteFileOnCloseReader in class ArchiveReader

Returns:

an ArchiveReader that will delete a local file on close. Used when we bring Archive files local and need to clean up afterward.

getDotFileExtension

public String getDotFileExtension()

Specified by:

getDotFileExtension in class ArchiveReader

getFileExtension

public String getFileExtension()

Specified by:

getFileExtension in class ArchiveReader

output

protected static void output(WARCReader reader,
 String format)
                      throws IOException,
ParseException

Write out the arcfile.

Parameters:

reader -

format - Format to use outputting.

Throws:

IOException

ParseException

createCDXIndexFile

public static void createCDXIndexFile(String urlOrPath)
                               throws IOException,
ParseException

Generate a CDX index file for an ARC file.

Parameters:

urlOrPath - The ARC file to generate a CDX index for

Throws:

IOException

ParseException

main

public static void main(String[] args)
                 throws org.apache.commons.cli.ParseException,
IOException,
ParseException

Command-line interface to WARCReader. Here is the command-line interface:

 usage: java org.archive.io.arc.WARCReader [--offset=#] ARCFILE
  -h,--help      Prints this message and exits.
  -o,--offset    Outputs record at this offset into arc file.

Outputs using a pseudo-CDX format as described here: CDX Legent and here Example. Legend used in below is: 'CDX b e a m s c V (or v if uncompressed) n g'. Hash is hard-coded straight SHA-1 hash of content.

Parameters:

args - Command-line arguments.

Throws:

org.apache.commons.cli.ParseException - Failed parse of the command line.

IOException

ParseException