WARCConstants (webarchive-commons 1.1.5 API)

All Superinterfaces:

ArchiveFileConstants

All Known Subinterfaces:

WARCConstants

All Known Implementing Classes:

WARCReader, WARCReaderFactory, WARCReaderFactory.CompressedWARCReader, WARCReaderFactory.UncompressedWARCReader, WARCRecord, WARCRecordWriter, WARCWriter
```
public interface WARCConstants
extends ArchiveFileConstants
```
WARC Constants used by WARC readers and writers.

Nested Class Summary

Nested Classes
Modifier and Type Interface and Description

static class WARCConstants.WARCRecordType
WARC Record Types.

Nested Classes
Modifier and Type	Interface and Description
`static class`	`WARCConstants.WARCRecordType` WARC Record Types.

Field Summary

Fields
Modifier and Type	Field and Description
`static String`	`COLON_SPACE`
`static String`	`COMPRESSED_WARC_FILE_EXTENSION` Compressed WARC file extension.
`static String`	`CONTENT_DESCRIPTION`
`static String`	`CONTENT_LENGTH`
`static String`	`CONTENT_TYPE`
`static String`	`DEFAULT_ENCODING` Encoding to use getting bytes from strings.
`static int`	`DEFAULT_MAX_WARC_FILE_SIZE` Default maximum WARC file size.
`static String`	`DOT_COMPRESSED_WARC_FILE_EXTENSION` Compressed dot WARC file extension.
`static String`	`DOT_WARC_FILE_EXTENSION` Dot WARC file extension.
`static String`	`FTP_CONTROL_CONVERSATION_MIMETYPE`
`static char`	`HEADER_FIELD_SEPARATOR` Header field seperator character.
`static String`	`HEADER_KEY_BLOCK_DIGEST`
`static String`	`HEADER_KEY_CONCURRENT_TO`
`static String`	`HEADER_KEY_DATE`
`static String`	`HEADER_KEY_ETAG`
`static String`	`HEADER_KEY_FILENAME`
`static String`	`HEADER_KEY_ID`
`static String`	`HEADER_KEY_IP`
`static String`	`HEADER_KEY_LAST_MODIFIED`
`static String`	`HEADER_KEY_PAYLOAD_DIGEST`
`static String`	`HEADER_KEY_PROFILE`
`static String`	`HEADER_KEY_REFERS_TO`
`static String`	`HEADER_KEY_REFERS_TO_DATE`
`static String`	`HEADER_KEY_REFERS_TO_FILE_OFFSET`
`static String`	`HEADER_KEY_REFERS_TO_FILENAME`
`static String`	`HEADER_KEY_REFERS_TO_TARGET_URI` These fields help a consumer of the warc to locate the warc record that "WARC-Refers-To" refers to.
`static String`	`HEADER_KEY_TRUNCATED`
`static String`	`HEADER_KEY_TYPE`
`static String`	`HEADER_KEY_URI`
`static String`	`HEADER_LINE_ENCODING`
`static String`	`HTTP_REQUEST_MIMETYPE` To be safe, lets use application type rather than message.
`static String`	`HTTP_RESPONSE_MIMETYPE`
`static int`	`MAX_LINE_LENGTH`
`static int`	`MAX_WARC_HEADER_LINE_LENGTH` Assumed maximum size of a Header Line.
`static String`	`NAMED_FIELD_CHECKSUM_LABEL`
`static String`	`NAMED_FIELD_DESCRIPTION`
`static String`	`NAMED_FIELD_FILEDESC`
`static String`	`NAMED_FIELD_IP_LABEL`
`static String`	`NAMED_FIELD_RELATED_LABEL`
`static String`	`NAMED_FIELD_TRUNCATED`
`static String`	`NAMED_FIELD_TRUNCATED_VALUE_HEAD`
`static String`	`NAMED_FIELD_TRUNCATED_VALUE_LENGTH`
`static String`	`NAMED_FIELD_TRUNCATED_VALUE_TIME`
`static String`	`NAMED_FIELD_TRUNCATED_VALUE_UNSPECIFIED`
`static String`	`NAMED_FIELD_WARCFILENAME`
`static String`	`PLACEHOLDER_RECORD_LENGTH_STRING` Placeholder for length in Header line.
`static String`	`PROFILE_REVISIT_IDENTICAL_DIGEST`
`static String`	`PROFILE_REVISIT_NOT_MODIFIED`
`static String`	`PROFILE_REVISIT_URI_AGNOSTIC_IDENTICAL_DIGEST`
`static String`	`TRUNCATED_VALUE_UNSPECIFIED`
`static String`	`TYPE`
`static String`	`WARC_FIELDS_TYPE`
`static String`	`WARC_FILE_EXTENSION` WARC file extention.
`static String`	`WARC_HEADER_ENCODING`
`static String`	`WARC_ID` WARC-ID
`static String`	`WARC_MAGIC` WARC MAGIC WARC files and records begin with this sequence.
`static String`	`WARC_VERSION` Hard-coded version for WARC files made with this code.
`static Character[]`	`WSP` WSP One of a space or horizontal tab character.

Fields inherited from interface org.archive.format.ArchiveFileConstants
ABSOLUTE_OFFSET_KEY, CDX, CDX_FILE, CDX_LINE_BUFFER_SIZE, CRLF, DATE_FIELD_KEY, DEFAULT_DIGEST_METHOD, DOT_COMPRESSED_FILE_EXTENSION, DUMP, GZIP_DUMP, HEADER, INVALID_SUFFIX, LENGTH_FIELD_KEY, MIMETYPE_FIELD_KEY, NOHEAD, OCCUPIED_SUFFIX, READER_IDENTIFIER_FIELD_KEY, RECORD_IDENTIFIER_FIELD_KEY, SINGLE_SPACE, TYPE_FIELD_KEY, URL_FIELD_KEY, VERSION_FIELD_KEY

- Field Detail
  - DEFAULT_MAX_WARC_FILE_SIZE
```
static final int DEFAULT_MAX_WARC_FILE_SIZE
```
    Default maximum WARC file size. 1Gig.
    
    See Also:
    Constant Field Values
  - WARC_MAGIC
```
static final String WARC_MAGIC
```
    WARC MAGIC WARC files and records begin with this sequence.
    
    See Also:
    Constant Field Values
  - WARC_VERSION
```
static final String WARC_VERSION
```
    Hard-coded version for WARC files made with this code. conforms to ISO 28500:2009 as of May 2009
    
    See Also:
    Constant Field Values
  - MAX_WARC_HEADER_LINE_LENGTH
```
static final int MAX_WARC_HEADER_LINE_LENGTH
```
    Assumed maximum size of a Header Line. This 100k which seems massive but its the same as the LINE_LENGTH from alexa/include/a_arcio.h:
```
 #define LINE_LENGTH     (100*1024)
 
```
    See Also:
    Constant Field Values
  - MAX_LINE_LENGTH
```
static final int MAX_LINE_LENGTH
```
    See Also:
    Constant Field Values
  - WARC_FILE_EXTENSION
```
static final String WARC_FILE_EXTENSION
```
    WARC file extention.
    
    See Also:
    Constant Field Values
  - DOT_WARC_FILE_EXTENSION
```
static final String DOT_WARC_FILE_EXTENSION
```
    Dot WARC file extension.
    
    See Also:
    Constant Field Values
  - COMPRESSED_WARC_FILE_EXTENSION
```
static final String COMPRESSED_WARC_FILE_EXTENSION
```
    Compressed WARC file extension.
    
    See Also:
    Constant Field Values
  - DOT_COMPRESSED_WARC_FILE_EXTENSION
```
static final String DOT_COMPRESSED_WARC_FILE_EXTENSION
```
    Compressed dot WARC file extension.
    
    See Also:
    Constant Field Values
  - DEFAULT_ENCODING
```
static final String DEFAULT_ENCODING
```
    Encoding to use getting bytes from strings. Specify an encoding rather than leave it to chance: i.e whatever the JVMs encoding. Use an encoding that gets the stream as bytes, not chars.
    TODO: ARC uses ISO-8859-1. In general, we should use UTF-8 but we probably need a single byte encoding if we're out for preserving the binary data as received over the net (We probably don't want to transform the supra-ASCII characters to UTF-8 before storing in ARC). For now, till we figure it, DEFAULT_ENCODING is single-byte charset -- same as ARCs.
    
    See Also:
    Constant Field Values
  - HEADER_LINE_ENCODING
```
static final String HEADER_LINE_ENCODING
```
    See Also:
    Constant Field Values
  - WARC_HEADER_ENCODING
```
static final String WARC_HEADER_ENCODING
```
    See Also:
    Constant Field Values
  - TYPE
```
static final String TYPE
```
    See Also:
    Constant Field Values
  - WARC_ID
```
static final String WARC_ID
```
    WARC-ID
    
    See Also:
    Constant Field Values
  - HEADER_FIELD_SEPARATOR
```
static final char HEADER_FIELD_SEPARATOR
```
    Header field seperator character.
    
    See Also:
    Constant Field Values
  - WSP
```
static final Character[] WSP
```
    WSP One of a space or horizontal tab character. TODO: WSP undefined. Fix.
  - PLACEHOLDER_RECORD_LENGTH_STRING
```
static final String PLACEHOLDER_RECORD_LENGTH_STRING
```
    Placeholder for length in Header line. Placeholder is same size as the fixed field size allocated for length, 12 characters. 12 characters allows records of size almost 1TB.
    
    See Also:
    Constant Field Values
  - NAMED_FIELD_IP_LABEL
```
static final String NAMED_FIELD_IP_LABEL
```
    See Also:
    Constant Field Values
  - NAMED_FIELD_CHECKSUM_LABEL
```
static final String NAMED_FIELD_CHECKSUM_LABEL
```
    See Also:
    Constant Field Values
  - NAMED_FIELD_RELATED_LABEL
```
static final String NAMED_FIELD_RELATED_LABEL
```
    See Also:
    Constant Field Values
  - NAMED_FIELD_WARCFILENAME
```
static final String NAMED_FIELD_WARCFILENAME
```
    See Also:
    Constant Field Values
  - NAMED_FIELD_DESCRIPTION
```
static final String NAMED_FIELD_DESCRIPTION
```
    See Also:
    Constant Field Values
  - NAMED_FIELD_FILEDESC
```
static final String NAMED_FIELD_FILEDESC
```
    See Also:
    Constant Field Values
  - NAMED_FIELD_TRUNCATED
```
static final String NAMED_FIELD_TRUNCATED
```
    See Also:
    Constant Field Values
  - NAMED_FIELD_TRUNCATED_VALUE_TIME
```
static final String NAMED_FIELD_TRUNCATED_VALUE_TIME
```
    See Also:
    Constant Field Values
  - NAMED_FIELD_TRUNCATED_VALUE_LENGTH
```
static final String NAMED_FIELD_TRUNCATED_VALUE_LENGTH
```
    See Also:
    Constant Field Values
  - NAMED_FIELD_TRUNCATED_VALUE_HEAD
```
static final String NAMED_FIELD_TRUNCATED_VALUE_HEAD
```
    See Also:
    Constant Field Values
  - NAMED_FIELD_TRUNCATED_VALUE_UNSPECIFIED
```
static final String NAMED_FIELD_TRUNCATED_VALUE_UNSPECIFIED
```
  - HEADER_KEY_URI
```
static final String HEADER_KEY_URI
```
    See Also:
    Constant Field Values
  - HEADER_KEY_DATE
```
static final String HEADER_KEY_DATE
```
    See Also:
    Constant Field Values
  - HEADER_KEY_TYPE
```
static final String HEADER_KEY_TYPE
```
    See Also:
    Constant Field Values
  - HEADER_KEY_ID
```
static final String HEADER_KEY_ID
```
    See Also:
    Constant Field Values
  - HEADER_KEY_IP
```
static final String HEADER_KEY_IP
```
    See Also:
    Constant Field Values
  - HEADER_KEY_BLOCK_DIGEST
```
static final String HEADER_KEY_BLOCK_DIGEST
```
    See Also:
    Constant Field Values
  - HEADER_KEY_PAYLOAD_DIGEST
```
static final String HEADER_KEY_PAYLOAD_DIGEST
```
    See Also:
    Constant Field Values
  - HEADER_KEY_CONCURRENT_TO
```
static final String HEADER_KEY_CONCURRENT_TO
```
    See Also:
    Constant Field Values
  - HEADER_KEY_TRUNCATED
```
static final String HEADER_KEY_TRUNCATED
```
    See Also:
    Constant Field Values
  - HEADER_KEY_PROFILE
```
static final String HEADER_KEY_PROFILE
```
    See Also:
    Constant Field Values
  - HEADER_KEY_FILENAME
```
static final String HEADER_KEY_FILENAME
```
    See Also:
    Constant Field Values
  - HEADER_KEY_ETAG
```
static final String HEADER_KEY_ETAG
```
    See Also:
    Constant Field Values
  - HEADER_KEY_LAST_MODIFIED
```
static final String HEADER_KEY_LAST_MODIFIED
```
    See Also:
    Constant Field Values
  - HEADER_KEY_REFERS_TO
```
static final String HEADER_KEY_REFERS_TO
```
    See Also:
    Constant Field Values
  - HEADER_KEY_REFERS_TO_TARGET_URI
```
static final String HEADER_KEY_REFERS_TO_TARGET_URI
```
    These fields help a consumer of the warc to locate the warc record that "WARC-Refers-To" refers to.
    
    See Also:
    WARCWriterProcessor, Constant Field Values
  - HEADER_KEY_REFERS_TO_DATE
```
static final String HEADER_KEY_REFERS_TO_DATE
```
    See Also:
    Constant Field Values
  - HEADER_KEY_REFERS_TO_FILENAME
```
static final String HEADER_KEY_REFERS_TO_FILENAME
```
    See Also:
    Constant Field Values
  - HEADER_KEY_REFERS_TO_FILE_OFFSET
```
static final String HEADER_KEY_REFERS_TO_FILE_OFFSET
```
    See Also:
    Constant Field Values
  - PROFILE_REVISIT_URI_AGNOSTIC_IDENTICAL_DIGEST
```
static final String PROFILE_REVISIT_URI_AGNOSTIC_IDENTICAL_DIGEST
```
    See Also:
    Constant Field Values
  - PROFILE_REVISIT_IDENTICAL_DIGEST
```
static final String PROFILE_REVISIT_IDENTICAL_DIGEST
```
    See Also:
    Constant Field Values
  - PROFILE_REVISIT_NOT_MODIFIED
```
static final String PROFILE_REVISIT_NOT_MODIFIED
```
    See Also:
    Constant Field Values
  - CONTENT_LENGTH
```
static final String CONTENT_LENGTH
```
    See Also:
    Constant Field Values
  - CONTENT_TYPE
```
static final String CONTENT_TYPE
```
    See Also:
    Constant Field Values
  - CONTENT_DESCRIPTION
```
static final String CONTENT_DESCRIPTION
```
    See Also:
    Constant Field Values
  - COLON_SPACE
```
static final String COLON_SPACE
```
    See Also:
    Constant Field Values
  - TRUNCATED_VALUE_UNSPECIFIED
```
static final String TRUNCATED_VALUE_UNSPECIFIED
```
    See Also:
    Constant Field Values
  - HTTP_REQUEST_MIMETYPE
```
static final String HTTP_REQUEST_MIMETYPE
```
    To be safe, lets use application type rather than message. Regards 'message/http', RFC says "...provided that it obeys the MIME restrictions for all 'message' types regarding line length and encodings." This usually means lines of 1000 octets max (unless a 'Content-Transfer-Encoding: binary' mime header is present).
    
    See Also:
    rfc2616 section 19.1, Constant Field Values
  - HTTP_RESPONSE_MIMETYPE
```
static final String HTTP_RESPONSE_MIMETYPE
```
    See Also:
    Constant Field Values
  - FTP_CONTROL_CONVERSATION_MIMETYPE
```
static final String FTP_CONTROL_CONVERSATION_MIMETYPE
```
    See Also:
    Constant Field Values
  - WARC_FIELDS_TYPE
```
static final String WARC_FIELDS_TYPE
```
    See Also:
    Constant Field Values

Interface WARCConstants

Nested Class Summary

Field Summary

Fields inherited from interface org.archive.format.ArchiveFileConstants

Field Detail

DEFAULT_MAX_WARC_FILE_SIZE

WARC_MAGIC

WARC_VERSION

MAX_WARC_HEADER_LINE_LENGTH

MAX_LINE_LENGTH

WARC_FILE_EXTENSION

DOT_WARC_FILE_EXTENSION

COMPRESSED_WARC_FILE_EXTENSION

DOT_COMPRESSED_WARC_FILE_EXTENSION

DEFAULT_ENCODING

HEADER_LINE_ENCODING

WARC_HEADER_ENCODING

TYPE

WARC_ID

HEADER_FIELD_SEPARATOR

WSP

PLACEHOLDER_RECORD_LENGTH_STRING

NAMED_FIELD_IP_LABEL

NAMED_FIELD_CHECKSUM_LABEL

NAMED_FIELD_RELATED_LABEL

NAMED_FIELD_WARCFILENAME

NAMED_FIELD_DESCRIPTION

NAMED_FIELD_FILEDESC

NAMED_FIELD_TRUNCATED

NAMED_FIELD_TRUNCATED_VALUE_TIME

NAMED_FIELD_TRUNCATED_VALUE_LENGTH

NAMED_FIELD_TRUNCATED_VALUE_HEAD

NAMED_FIELD_TRUNCATED_VALUE_UNSPECIFIED

HEADER_KEY_URI

HEADER_KEY_DATE

HEADER_KEY_TYPE

HEADER_KEY_ID

HEADER_KEY_IP

HEADER_KEY_BLOCK_DIGEST

HEADER_KEY_PAYLOAD_DIGEST

HEADER_KEY_CONCURRENT_TO

HEADER_KEY_TRUNCATED

HEADER_KEY_PROFILE

HEADER_KEY_FILENAME

HEADER_KEY_ETAG

HEADER_KEY_LAST_MODIFIED

HEADER_KEY_REFERS_TO

HEADER_KEY_REFERS_TO_TARGET_URI

HEADER_KEY_REFERS_TO_DATE

HEADER_KEY_REFERS_TO_FILENAME

HEADER_KEY_REFERS_TO_FILE_OFFSET

PROFILE_REVISIT_URI_AGNOSTIC_IDENTICAL_DIGEST

PROFILE_REVISIT_IDENTICAL_DIGEST

PROFILE_REVISIT_NOT_MODIFIED

CONTENT_LENGTH

CONTENT_TYPE

CONTENT_DESCRIPTION

COLON_SPACE

TRUNCATED_VALUE_UNSPECIFIED

HTTP_REQUEST_MIMETYPE

HTTP_RESPONSE_MIMETYPE

FTP_CONTROL_CONVERSATION_MIMETYPE

WARC_FIELDS_TYPE