org.archive.io.arc

Class ARCReaderFactory

java.lang.Object

org.archive.io.ArchiveReaderFactory

org.archive.io.arc.ARCReaderFactory

All Implemented Interfaces:

ARCConstants, ArchiveFileConstants

public class ARCReaderFactory
extends ArchiveReaderFactory
implements ARCConstants

Factory that returns an ARCReader. Can handle compressed and uncompressed ARCs.

Author:

stack

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	ARCReaderFactory.CompressedARCReader Compressed arc file reader.
class	ARCReaderFactory.UncompressedARCReader Uncompressed arc file reader.

Nested classes/interfaces inherited from interface org.archive.format.arc.ARCConstants

ARCConstants.ArcRecordErrors

Field Summary

Fields inherited from interface org.archive.format.arc.ARCConstants

ACTUAL_LENGTH_KEY, ALEXA_DAT_MIME, ARC_FILE_EXTENSION, ARC_GZIP_EXTRA_FIELD, ARC_MAGIC_NUMBER, ARC_META_CHARSET, CARRIAGE_RETURN_ORD, CHECKSUM_FIELD_KEY, CHECKSUM_HEADER_FIELD_KEY, CODE_HEADER_FIELD_KEY, COMPRESSED_ARC_FILE_EXTENSION, DATE_STRING_KEY, DECLARED_LENGTH_KEY, DEFAULT_ENCODING, DEFAULT_GZIP_HEADER_LENGTH, DEFAULT_MAX_ARC_FILE_SIZE, DEFAULT_MIME, DELIMITER, DNS_MIME, DOT_ARC_FILE_EXTENSION, DOT_COMPRESSED_ARC_FILE_EXTENSION, DOT_COMPRESSED_FILE_EXTENSION, FIELD_COUNT, FILEDESC_SCHEME, FILENAME_FIELD_KEY, FILENAME_HEADER_FIELD_KEY, GZIP_HEADER_BEGIN, HEADER_FIELD_SEPARATOR, HEADER_LENGTH, IP_HEADER_FIELD_KEY, IP_KEY, LEADING_NEWLINES_KEY, LINE_SEPARATOR, LOCATION_HEADER_FIELD_KEY, MAX_HEADER_LINE_LENGTH, MAX_META_LENGTH, MAX_METADATA_LINE_LENGTH, MIME_KEY, MINIMUM_RECORD_LENGTH, NEW_LINE_ORD, OFFSET_FIELD_KEY, OFFSET_HEADER_FIELD_KEY, REQUIRED_VERSION_1_HEADER_FIELDS, STATUSCODE_FIELD_KEY, TOKENIZED_PREFIX, TRAILING_CRUFT_KEY, TRAILING_NEWLINES_KEY, URL_KEY

Fields inherited from interface org.archive.format.ArchiveFileConstants

ABSOLUTE_OFFSET_KEY, CDX, CDX_FILE, CDX_LINE_BUFFER_SIZE, CRLF, DATE_FIELD_KEY, DEFAULT_DIGEST_METHOD, DUMP, GZIP_DUMP, HEADER, INVALID_SUFFIX, LENGTH_FIELD_KEY, MIMETYPE_FIELD_KEY, NOHEAD, OCCUPIED_SUFFIX, READER_IDENTIFIER_FIELD_KEY, RECORD_IDENTIFIER_FIELD_KEY, SINGLE_SPACE, TYPE_FIELD_KEY, URL_FIELD_KEY, VERSION_FIELD_KEY

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	ARCReaderFactory() Shutdown any access to default constructor.

Method Summary

Methods

Modifier and Type	Method and Description
static ARCReader	get(File f)
static ARCReader	get(File f, boolean skipSuffixTest, long offset)
static ARCReader	get(File f, long offset)
static ARCReader	get(String arcFileOrUrl)
static ArchiveReader	get(String s, InputStream is, boolean atFirstRecord)
static ARCReader	get(String arcFileOrUrl, long offset)
static ARCReader	get(URL arcUrl) Get an ARCReader.
static ARCReader	get(URL arcUrl, long offset) Get an ARCReader aligned at offset.
protected ArchiveReader	getArchiveReader(File arcFile, boolean skipSuffixTest, long offset)
protected ArchiveReader	getArchiveReader(File f, long offset)
protected ArchiveReader	getArchiveReader(String arc, InputStream is, boolean atFirstRecord)
static boolean	isARCSuffix(String arcName)
boolean	isCompressed(File arcFile)
static boolean	testCompressedARCFile(File arcFile) Check file is compressed and in ARC GZIP format.
static boolean	testCompressedARCFile(File arcFile, boolean skipSuffixCheck) Check file is compressed and in ARC GZIP format.
static boolean	testCompressedARCStream(InputStream is) Tests passed stream is gzip stream by reading in the HEAD.

Methods inherited from class org.archive.io.ArchiveReaderFactory

addUserAgent, getArchiveReader, getArchiveReader, getArchiveReader, getArchiveReader, getArchiveReader, makeARCLocal

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ARCReaderFactory

protected ARCReaderFactory()

Shutdown any access to default constructor.

Method Detail

get

public static ARCReader get(String arcFileOrUrl)
                     throws MalformedURLException,
                            IOException

Throws:

MalformedURLException

IOException

get

public static ARCReader get(String arcFileOrUrl,
            long offset)
                     throws MalformedURLException,
                            IOException

Throws:

MalformedURLException

IOException

get

public static ARCReader get(File f)
                     throws IOException

Throws:

IOException

get

public static ARCReader get(File f,
            long offset)
                     throws IOException

Throws:

IOException

getArchiveReader

protected ArchiveReader getArchiveReader(File f,
                             long offset)
                                  throws IOException

Overrides:

getArchiveReader in class ArchiveReaderFactory

Throws:

IOException

get

public static ARCReader get(File f,
            boolean skipSuffixTest,
            long offset)
                     throws IOException

Parameters:

f - An arcfile to read.

skipSuffixTest - Set to true if want to test that ARC has proper suffix. Use this method and pass false to open ARCs with the .open or otherwise suffix.

offset - Have returned ARCReader set to start reading at passed offset.

Returns:

An ARCReader.

Throws:

IOException

getArchiveReader

protected ArchiveReader getArchiveReader(File arcFile,
                             boolean skipSuffixTest,
                             long offset)
                                  throws IOException

Throws:

IOException

get

public static ArchiveReader get(String s,
                InputStream is,
                boolean atFirstRecord)
                         throws IOException

Throws:

IOException

getArchiveReader

protected ArchiveReader getArchiveReader(String arc,
                             InputStream is,
                             boolean atFirstRecord)
                                  throws IOException

Overrides:

getArchiveReader in class ArchiveReaderFactory

Throws:

IOException

get

public static ARCReader get(URL arcUrl,
            long offset)
                     throws IOException

Get an ARCReader aligned at offset. This version of get will not bring the ARC local but will try to stream across the net making an HTTP 1.1 Range request on remote http server (RFC1435 Section 14.35).

Parameters:

arcUrl - HTTP URL for an ARC (All ARCs considered remote).

offset - Offset into ARC at which to start fetching.

Returns:

An ARCReader aligned at offset.

Throws:

IOException

get

public static ARCReader get(URL arcUrl)
                     throws IOException

Get an ARCReader. Pulls the ARC local into whereever the System Property java.io.tmpdir points. It then hands back an ARCReader that points at this local copy. A close on this ARCReader instance will remove the local copy.

Parameters:

arcUrl - An URL that points at an ARC.

Returns:

An ARCReader.

Throws:

IOException

isCompressed

public boolean isCompressed(File arcFile)
                     throws IOException

Overrides:

isCompressed in class ArchiveReaderFactory

Parameters:

arcFile - File to test.

Returns:

True if arcFile is compressed ARC.

Throws:

IOException

testCompressedARCFile

public static boolean testCompressedARCFile(File arcFile)
                                     throws IOException

Check file is compressed and in ARC GZIP format.

Parameters:

arcFile - File to test if its Internet Archive ARC file GZIP compressed.

Returns:

True if this is an Internet Archive GZIP'd ARC file (It begins w/ the Internet Archive GZIP header and has the COMPRESSED_ARC_FILE_EXTENSION suffix).

Throws:

IOException - If file does not exist or is not unreadable.

testCompressedARCFile

public static boolean testCompressedARCFile(File arcFile,
                            boolean skipSuffixCheck)
                                     throws IOException

Check file is compressed and in ARC GZIP format.

Parameters:

arcFile - File to test if its Internet Archive ARC file GZIP compressed.

skipSuffixCheck - Set to true if we're not to test on the '.arc.gz' suffix.

Returns:

True if this is an Internet Archive GZIP'd ARC file (It begins w/ the Internet Archive GZIP header).

Throws:

IOException - If file does not exist or is not unreadable.

isARCSuffix

public static boolean isARCSuffix(String arcName)

testCompressedARCStream

public static boolean testCompressedARCStream(InputStream is)
                                       throws IOException

Tests passed stream is gzip stream by reading in the HEAD. Does not reposition the stream. That is left up to the caller.

Parameters:

is - An InputStream.

Returns:

True if compressed stream.

Throws:

IOException