Interface WARCConstants

All Superinterfaces:

ArchiveFileConstants

All Known Implementing Classes:

WARCRecordWriter

public interface WARCConstants
extends ArchiveFileConstants

WARC Constants used by WARC readers and writers.

Author:

stack

Nested Class Summary

Nested Classes

Modifier and Type	Interface	Description
static enum	WARCConstants.WARCRecordType	WARC Record Types.

Field Summary

Fields

Modifier and Type	Field	Description
static final String	COLON_SPACE
static final String	COMPRESSED_WARC_FILE_EXTENSION	Compressed WARC file extension.
static final String	CONTENT_DESCRIPTION
static final String	CONTENT_LENGTH
static final String	CONTENT_TYPE
static final String	DEFAULT_ENCODING	Encoding to use getting bytes from strings.
static final int	DEFAULT_MAX_WARC_FILE_SIZE	Default maximum WARC file size. 1Gig.
static final String	DOT_COMPRESSED_WARC_FILE_EXTENSION	Compressed dot WARC file extension.
static final String	DOT_WARC_FILE_EXTENSION	Dot WARC file extension.
static final String	FTP_CONTROL_CONVERSATION_MIMETYPE
static final char	HEADER_FIELD_SEPARATOR	Header field seperator character.
static final String	HEADER_KEY_BLOCK_DIGEST
static final String	HEADER_KEY_CONCURRENT_TO
static final String	HEADER_KEY_DATE
static final String	HEADER_KEY_ETAG
static final String	HEADER_KEY_FILENAME
static final String	HEADER_KEY_ID
static final String	HEADER_KEY_IP
static final String	HEADER_KEY_LAST_MODIFIED
static final String	HEADER_KEY_PAYLOAD_DIGEST
static final String	HEADER_KEY_PROFILE
static final String	HEADER_KEY_REFERS_TO
static final String	HEADER_KEY_REFERS_TO_DATE
static final String	HEADER_KEY_REFERS_TO_FILE_OFFSET
static final String	HEADER_KEY_REFERS_TO_FILENAME
static final String	HEADER_KEY_REFERS_TO_TARGET_URI	These fields help a consumer of the warc to locate the warc record that "WARC-Refers-To" refers to.
static final String	HEADER_KEY_TRUNCATED
static final String	HEADER_KEY_TYPE
static final String	HEADER_KEY_URI
static final String	HEADER_LINE_ENCODING
static final String	HTTP_REQUEST_MIMETYPE	To be safe, lets use application type rather than message.
static final String	HTTP_RESPONSE_MIMETYPE
static final String	HTTP_RESPONSE_MIMETYPE_NS
static final int	MAX_LINE_LENGTH
static final int	MAX_WARC_HEADER_LINE_LENGTH	Assumed maximum size of a Header Line.
static final String	NAMED_FIELD_CHECKSUM_LABEL
static final String	NAMED_FIELD_DESCRIPTION
static final String	NAMED_FIELD_FILEDESC
static final String	NAMED_FIELD_IP_LABEL
static final String	NAMED_FIELD_RELATED_LABEL
static final String	NAMED_FIELD_TRUNCATED
static final String	NAMED_FIELD_TRUNCATED_VALUE_HEAD
static final String	NAMED_FIELD_TRUNCATED_VALUE_LENGTH
static final String	NAMED_FIELD_TRUNCATED_VALUE_TIME
static final String	NAMED_FIELD_TRUNCATED_VALUE_UNSPECIFIED
static final String	NAMED_FIELD_WARCFILENAME
static final String	PLACEHOLDER_RECORD_LENGTH_STRING	Placeholder for length in Header line.
static final String	PROFILE_REVISIT_IDENTICAL_DIGEST
static final String	PROFILE_REVISIT_NOT_MODIFIED
static final String	TRUNCATED_VALUE_UNSPECIFIED
static final String	TYPE
static final String	WARC_FIELDS_TYPE
static final String	WARC_FILE_EXTENSION	WARC file extention.
static final String	WARC_HEADER_ENCODING
static final String	WARC_ID	WARC-ID
static final String	WARC_MAGIC	WARC MAGIC WARC files and records begin with this sequence.
static final String	WARC_VERSION	Hard-coded version for WARC files made with this code.
static final Character[]	WSP	WSP One of a space or horizontal tab character.

Fields inherited from interface org.archive.format.ArchiveFileConstants

ABSOLUTE_OFFSET_KEY, CDX, CDX_FILE, CDX_LINE_BUFFER_SIZE, CRLF, DATE_FIELD_KEY, DEFAULT_DIGEST_METHOD, DOT_COMPRESSED_FILE_EXTENSION, DUMP, GZIP_DUMP, HEADER, INVALID_SUFFIX, LENGTH_FIELD_KEY, MIMETYPE_FIELD_KEY, NOHEAD, OCCUPIED_SUFFIX, ORIGIN_FIELD_KEY, READER_IDENTIFIER_FIELD_KEY, RECORD_IDENTIFIER_FIELD_KEY, SINGLE_SPACE, TYPE_FIELD_KEY, URL_FIELD_KEY, VERSION_FIELD_KEY

Field Details

DEFAULT_MAX_WARC_FILE_SIZE

static final int DEFAULT_MAX_WARC_FILE_SIZE

Default maximum WARC file size. 1Gig.

See Also:

• Constant Field Values

WARC_MAGIC

static final String WARC_MAGIC

WARC MAGIC WARC files and records begin with this sequence.

See Also:

• Constant Field Values

WARC_VERSION

static final String WARC_VERSION

Hard-coded version for WARC files made with this code. conforms to ISO 28500:2009 as of May 2009

See Also:

• Constant Field Values

MAX_WARC_HEADER_LINE_LENGTH

static final int MAX_WARC_HEADER_LINE_LENGTH

Assumed maximum size of a Header Line. This 100k which seems massive but its the same as the LINE_LENGTH from alexa/include/a_arcio.h:

 #define LINE_LENGTH     (100*1024)
 

See Also:

• Constant Field Values

MAX_LINE_LENGTH

static final int MAX_LINE_LENGTH

See Also:

• Constant Field Values

WARC_FILE_EXTENSION

static final String WARC_FILE_EXTENSION

WARC file extention.

See Also:

• Constant Field Values

DOT_WARC_FILE_EXTENSION

static final String DOT_WARC_FILE_EXTENSION

Dot WARC file extension.

See Also:

• Constant Field Values

COMPRESSED_WARC_FILE_EXTENSION

static final String COMPRESSED_WARC_FILE_EXTENSION

Compressed WARC file extension.

See Also:

• Constant Field Values

DOT_COMPRESSED_WARC_FILE_EXTENSION

static final String DOT_COMPRESSED_WARC_FILE_EXTENSION

Compressed dot WARC file extension.

See Also:

• Constant Field Values

DEFAULT_ENCODING

static final String DEFAULT_ENCODING

Encoding to use getting bytes from strings. Specify an encoding rather than leave it to chance: i.e whatever the JVMs encoding. Use an encoding that gets the stream as bytes, not chars.

TODO: ARC uses ISO-8859-1. In general, we should use UTF-8 but we probably need a single byte encoding if we're out for preserving the binary data as received over the net (We probably don't want to transform the supra-ASCII characters to UTF-8 before storing in ARC). For now, till we figure it, DEFAULT_ENCODING is single-byte charset -- same as ARCs.

HEADER_LINE_ENCODING

static final String HEADER_LINE_ENCODING

WARC_HEADER_ENCODING

static final String WARC_HEADER_ENCODING

TYPE

static final String TYPE

See Also:

• Constant Field Values

WARC_ID

static final String WARC_ID

WARC-ID

See Also:

• Constant Field Values

HEADER_FIELD_SEPARATOR

static final char HEADER_FIELD_SEPARATOR

Header field seperator character.

See Also:

• Constant Field Values

WSP

static final Character[] WSP

WSP One of a space or horizontal tab character. TODO: WSP undefined. Fix.

PLACEHOLDER_RECORD_LENGTH_STRING

static final String PLACEHOLDER_RECORD_LENGTH_STRING

Placeholder for length in Header line. Placeholder is same size as the fixed field size allocated for length, 12 characters. 12 characters allows records of size almost 1TB.

See Also:

• Constant Field Values

NAMED_FIELD_IP_LABEL

static final String NAMED_FIELD_IP_LABEL

See Also:

• Constant Field Values

NAMED_FIELD_CHECKSUM_LABEL

static final String NAMED_FIELD_CHECKSUM_LABEL

See Also:

• Constant Field Values

NAMED_FIELD_RELATED_LABEL

static final String NAMED_FIELD_RELATED_LABEL

See Also:

• Constant Field Values

NAMED_FIELD_WARCFILENAME

static final String NAMED_FIELD_WARCFILENAME

See Also:

• Constant Field Values

NAMED_FIELD_DESCRIPTION

static final String NAMED_FIELD_DESCRIPTION

See Also:

• Constant Field Values

NAMED_FIELD_FILEDESC

static final String NAMED_FIELD_FILEDESC

See Also:

• Constant Field Values

NAMED_FIELD_TRUNCATED

static final String NAMED_FIELD_TRUNCATED

See Also:

• Constant Field Values

NAMED_FIELD_TRUNCATED_VALUE_TIME

static final String NAMED_FIELD_TRUNCATED_VALUE_TIME

See Also:

• Constant Field Values

NAMED_FIELD_TRUNCATED_VALUE_LENGTH

static final String NAMED_FIELD_TRUNCATED_VALUE_LENGTH

See Also:

• Constant Field Values

NAMED_FIELD_TRUNCATED_VALUE_HEAD

static final String NAMED_FIELD_TRUNCATED_VALUE_HEAD

See Also:

• Constant Field Values

NAMED_FIELD_TRUNCATED_VALUE_UNSPECIFIED

static final String NAMED_FIELD_TRUNCATED_VALUE_UNSPECIFIED

HEADER_KEY_URI

static final String HEADER_KEY_URI

See Also:

• Constant Field Values

HEADER_KEY_DATE

static final String HEADER_KEY_DATE

See Also:

• Constant Field Values

HEADER_KEY_TYPE

static final String HEADER_KEY_TYPE

See Also:

• Constant Field Values

HEADER_KEY_ID

static final String HEADER_KEY_ID

See Also:

• Constant Field Values

HEADER_KEY_IP

static final String HEADER_KEY_IP

See Also:

• Constant Field Values

HEADER_KEY_BLOCK_DIGEST

static final String HEADER_KEY_BLOCK_DIGEST

See Also:

• Constant Field Values

HEADER_KEY_PAYLOAD_DIGEST

static final String HEADER_KEY_PAYLOAD_DIGEST

See Also:

• Constant Field Values

HEADER_KEY_CONCURRENT_TO

static final String HEADER_KEY_CONCURRENT_TO

See Also:

• Constant Field Values

HEADER_KEY_TRUNCATED

static final String HEADER_KEY_TRUNCATED

See Also:

• Constant Field Values

HEADER_KEY_PROFILE

static final String HEADER_KEY_PROFILE

See Also:

• Constant Field Values

HEADER_KEY_FILENAME

static final String HEADER_KEY_FILENAME

See Also:

• Constant Field Values

HEADER_KEY_ETAG

static final String HEADER_KEY_ETAG

See Also:

• Constant Field Values

HEADER_KEY_LAST_MODIFIED

static final String HEADER_KEY_LAST_MODIFIED

See Also:

• Constant Field Values

HEADER_KEY_REFERS_TO

static final String HEADER_KEY_REFERS_TO

See Also:

• Constant Field Values

HEADER_KEY_REFERS_TO_TARGET_URI

static final String HEADER_KEY_REFERS_TO_TARGET_URI

These fields help a consumer of the warc to locate the warc record that "WARC-Refers-To" refers to.

See WARCWriterProcessor

See Also:

• Constant Field Values

HEADER_KEY_REFERS_TO_DATE

static final String HEADER_KEY_REFERS_TO_DATE

See Also:

• Constant Field Values

HEADER_KEY_REFERS_TO_FILENAME

static final String HEADER_KEY_REFERS_TO_FILENAME

See Also:

• Constant Field Values

HEADER_KEY_REFERS_TO_FILE_OFFSET

static final String HEADER_KEY_REFERS_TO_FILE_OFFSET

See Also:

• Constant Field Values

PROFILE_REVISIT_IDENTICAL_DIGEST

static final String PROFILE_REVISIT_IDENTICAL_DIGEST

See Also:

• Constant Field Values

PROFILE_REVISIT_NOT_MODIFIED

static final String PROFILE_REVISIT_NOT_MODIFIED

See Also:

• Constant Field Values

CONTENT_LENGTH

static final String CONTENT_LENGTH

See Also:

• Constant Field Values

CONTENT_TYPE

static final String CONTENT_TYPE

See Also:

• Constant Field Values

CONTENT_DESCRIPTION

static final String CONTENT_DESCRIPTION

See Also:

• Constant Field Values

COLON_SPACE

static final String COLON_SPACE

See Also:

• Constant Field Values

TRUNCATED_VALUE_UNSPECIFIED

static final String TRUNCATED_VALUE_UNSPECIFIED

See Also:

• Constant Field Values

HTTP_REQUEST_MIMETYPE

static final String HTTP_REQUEST_MIMETYPE

To be safe, lets use application type rather than message. Regards 'message/http', RFC says "...provided that it obeys the MIME restrictions for all 'message' types regarding line length and encodings." This usually means lines of 1000 octets max (unless a 'Content-Transfer-Encoding: binary' mime header is present).

See Also:

• rfc2616 section 19.1
• Constant Field Values

HTTP_RESPONSE_MIMETYPE

static final String HTTP_RESPONSE_MIMETYPE

See Also:

• Constant Field Values

HTTP_RESPONSE_MIMETYPE_NS

static final String HTTP_RESPONSE_MIMETYPE_NS

See Also:

• Constant Field Values

FTP_CONTROL_CONVERSATION_MIMETYPE

static final String FTP_CONTROL_CONVERSATION_MIMETYPE

See Also:

• Constant Field Values

WARC_FIELDS_TYPE

static final String WARC_FIELDS_TYPE

See Also:

• Constant Field Values