public class WARCReader extends ArchiveReader implements WARCConstants
WARCReaderFactory to get instance.ArchiveReader.ArchiveRecordIteratorWARCConstants.WARCRecordTypein, MAX_ALLOWED_RECOVERABLESCOLON_SPACE, COMPRESSED_WARC_FILE_EXTENSION, CONTENT_DESCRIPTION, CONTENT_LENGTH, CONTENT_TYPE, DEFAULT_ENCODING, DEFAULT_MAX_WARC_FILE_SIZE, DOT_COMPRESSED_WARC_FILE_EXTENSION, DOT_WARC_FILE_EXTENSION, FTP_CONTROL_CONVERSATION_MIMETYPE, HEADER_FIELD_SEPARATOR, HEADER_KEY_BLOCK_DIGEST, HEADER_KEY_CONCURRENT_TO, HEADER_KEY_DATE, HEADER_KEY_ETAG, HEADER_KEY_FILENAME, HEADER_KEY_ID, HEADER_KEY_IP, HEADER_KEY_LAST_MODIFIED, HEADER_KEY_PAYLOAD_DIGEST, HEADER_KEY_PROFILE, HEADER_KEY_REFERS_TO, HEADER_KEY_REFERS_TO_DATE, HEADER_KEY_REFERS_TO_FILE_OFFSET, HEADER_KEY_REFERS_TO_FILENAME, HEADER_KEY_REFERS_TO_TARGET_URI, HEADER_KEY_TRUNCATED, HEADER_KEY_TYPE, HEADER_KEY_URI, HEADER_LINE_ENCODING, HTTP_REQUEST_MIMETYPE, HTTP_RESPONSE_MIMETYPE, MAX_LINE_LENGTH, MAX_WARC_HEADER_LINE_LENGTH, NAMED_FIELD_CHECKSUM_LABEL, NAMED_FIELD_DESCRIPTION, NAMED_FIELD_FILEDESC, NAMED_FIELD_IP_LABEL, NAMED_FIELD_RELATED_LABEL, NAMED_FIELD_TRUNCATED, NAMED_FIELD_TRUNCATED_VALUE_HEAD, NAMED_FIELD_TRUNCATED_VALUE_LENGTH, NAMED_FIELD_TRUNCATED_VALUE_TIME, NAMED_FIELD_TRUNCATED_VALUE_UNSPECIFIED, NAMED_FIELD_WARCFILENAME, PLACEHOLDER_RECORD_LENGTH_STRING, PROFILE_REVISIT_IDENTICAL_DIGEST, PROFILE_REVISIT_NOT_MODIFIED, PROFILE_REVISIT_URI_AGNOSTIC_IDENTICAL_DIGEST, TRUNCATED_VALUE_UNSPECIFIED, TYPE, WARC_FIELDS_TYPE, WARC_FILE_EXTENSION, WARC_HEADER_ENCODING, WARC_ID, WARC_MAGIC, WARC_VERSION, WSPABSOLUTE_OFFSET_KEY, CDX, CDX_FILE, CDX_LINE_BUFFER_SIZE, CRLF, DATE_FIELD_KEY, DEFAULT_DIGEST_METHOD, DOT_COMPRESSED_FILE_EXTENSION, DUMP, GZIP_DUMP, HEADER, INVALID_SUFFIX, LENGTH_FIELD_KEY, MIMETYPE_FIELD_KEY, NOHEAD, OCCUPIED_SUFFIX, READER_IDENTIFIER_FIELD_KEY, RECORD_IDENTIFIER_FIELD_KEY, SINGLE_SPACE, TYPE_FIELD_KEY, URL_FIELD_KEY, VERSION_FIELD_KEY| Modifier | Constructor and Description |
|---|---|
protected |
WARCReader() |
| Modifier and Type | Method and Description |
|---|---|
protected WARCRecord |
createArchiveRecord(InputStream is,
long offset)
Create new WARC record.
|
static void |
createCDXIndexFile(String urlOrPath)
Generate a CDX index file for an ARC file.
|
void |
dump(boolean compress)
Dump this file on STDOUT
|
ArchiveReader |
getDeleteFileOnCloseReader(File f) |
String |
getDotFileExtension() |
String |
getFileExtension() |
protected void |
gotoEOR(ArchiveRecord record)
Skip over any trailing new lines at end of the record so we're lined up
ready to read the next.
|
protected void |
initialize(String i)
Convenience method used by subclass constructors.
|
static void |
main(String[] args)
Command-line interface to WARCReader.
|
protected static void |
output(WARCReader reader,
String format)
Write out the arcfile.
|
protected void |
readExpectedChar(InputStream is,
int expected) |
cdxOutput, cleanupCurrentRecord, close, currentRecord, get, get, getCurrentRecord, getFileName, getIn, getInputStream, getLogger, getOptions, getReaderIdentifier, getStrippedFileName, getStrippedFileName, getTrueOrFalse, getVersion, isCompressed, isDigest, isStrict, isValid, iterator, logStdErr, output, outputRecord, outputRecord, positionForRecord, setCompressed, setDigest, setIn, setReaderIdentifier, setStrict, setVersion, stripExtension, validate, validateprotected void initialize(String i)
ArchiveReaderinitialize in class ArchiveReaderi - Identifier for Archive file this reader goes against.protected void gotoEOR(ArchiveRecord record) throws IOException
gotoEOR in class ArchiveReaderrecord - IOExceptionprotected void readExpectedChar(InputStream is, int expected) throws IOException
IOExceptionprotected WARCRecord createArchiveRecord(InputStream is, long offset) throws IOException
createArchiveRecord in class ArchiveReaderis - InputStream to use.offset - Absolute offset into WARC file.IOExceptionpublic void dump(boolean compress)
throws IOException,
ParseException
ArchiveReaderdump in class ArchiveReaderIOExceptionParseExceptionpublic ArchiveReader getDeleteFileOnCloseReader(File f)
getDeleteFileOnCloseReader in class ArchiveReaderpublic String getDotFileExtension()
getDotFileExtension in class ArchiveReaderpublic String getFileExtension()
getFileExtension in class ArchiveReaderprotected static void output(WARCReader reader, String format) throws IOException, ParseException
reader - format - Format to use outputting.IOExceptionParseExceptionpublic static void createCDXIndexFile(String urlOrPath) throws IOException, ParseException
urlOrPath - The ARC file to generate a CDX index forIOExceptionParseExceptionpublic static void main(String[] args) throws org.apache.commons.cli.ParseException, IOException, ParseException
usage: java org.archive.io.arc.WARCReader [--offset=#] ARCFILE -h,--help Prints this message and exits. -o,--offset Outputs record at this offset into arc file.
Outputs using a pseudo-CDX format as described here: CDX Legent and here Example. Legend used in below is: 'CDX b e a m s c V (or v if uncompressed) n g'. Hash is hard-coded straight SHA-1 hash of content.
args - Command-line arguments.org.apache.commons.cli.ParseException - Failed parse of the command line.IOExceptionParseExceptionCopyright © 2015 The International Internet Preservation Consortium. All Rights Reserved.