org.archive.io.warc

Class WARCWriter

java.lang.Object

org.archive.io.WriterPoolMember

org.archive.io.warc.WARCWriter

All Implemented Interfaces:

ArchiveFileConstants, WARCConstants

public class WARCWriter
extends WriterPoolMember
implements WARCConstants

WARC implementation.

Assumption is that the caller is managing access to this WARCWriter ensuring only one thread accessing this WARC instance at any one time.

While being written, WARCs have a '.open' suffix appended.

Version:

$Revision: 4604 $ $Date: 2006-09-05 22:38:18 -0700 (Tue, 05 Sep 2006) $

Nested Class Summary

Nested classes/interfaces inherited from interface org.archive.format.warc.WARCConstants

WARCConstants.WARCRecordType

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	CONTENT_BYTES
static byte[]	CRLF_BYTES NEWLINE as bytes.
static java.lang.String	NUM_RECORDS
static java.lang.String	SIZE_ON_DISK
static java.lang.String	TOTAL_BYTES
static java.lang.String	TOTALS

Fields inherited from class org.archive.io.WriterPoolMember

countOut, currentBasename, currentTimestamp, DEFAULT_PREFIX, DEFAULT_TEMPLATE, f, out, roundRobinIndex, scratchbuffer, serialNoFormatter, settings, UTF8

Fields inherited from interface org.archive.format.warc.WARCConstants

COLON_SPACE, COMPRESSED_WARC_FILE_EXTENSION, CONTENT_DESCRIPTION, CONTENT_LENGTH, CONTENT_TYPE, DEFAULT_ENCODING, DEFAULT_MAX_WARC_FILE_SIZE, DOT_COMPRESSED_WARC_FILE_EXTENSION, DOT_WARC_FILE_EXTENSION, FTP_CONTROL_CONVERSATION_MIMETYPE, HEADER_FIELD_SEPARATOR, HEADER_KEY_BLOCK_DIGEST, HEADER_KEY_CONCURRENT_TO, HEADER_KEY_DATE, HEADER_KEY_ETAG, HEADER_KEY_FILENAME, HEADER_KEY_ID, HEADER_KEY_IP, HEADER_KEY_LAST_MODIFIED, HEADER_KEY_PAYLOAD_DIGEST, HEADER_KEY_PROFILE, HEADER_KEY_REFERS_TO, HEADER_KEY_REFERS_TO_DATE, HEADER_KEY_REFERS_TO_FILE_OFFSET, HEADER_KEY_REFERS_TO_FILENAME, HEADER_KEY_REFERS_TO_TARGET_URI, HEADER_KEY_TRUNCATED, HEADER_KEY_TYPE, HEADER_KEY_URI, HEADER_LINE_ENCODING, HTTP_REQUEST_MIMETYPE, HTTP_RESPONSE_MIMETYPE, HTTP_RESPONSE_MIMETYPE_NS, MAX_LINE_LENGTH, MAX_WARC_HEADER_LINE_LENGTH, NAMED_FIELD_CHECKSUM_LABEL, NAMED_FIELD_DESCRIPTION, NAMED_FIELD_FILEDESC, NAMED_FIELD_IP_LABEL, NAMED_FIELD_RELATED_LABEL, NAMED_FIELD_TRUNCATED, NAMED_FIELD_TRUNCATED_VALUE_HEAD, NAMED_FIELD_TRUNCATED_VALUE_LENGTH, NAMED_FIELD_TRUNCATED_VALUE_TIME, NAMED_FIELD_TRUNCATED_VALUE_UNSPECIFIED, NAMED_FIELD_WARCFILENAME, PLACEHOLDER_RECORD_LENGTH_STRING, PROFILE_REVISIT_IDENTICAL_DIGEST, PROFILE_REVISIT_NOT_MODIFIED, TRUNCATED_VALUE_UNSPECIFIED, TYPE, WARC_FIELDS_TYPE, WARC_FILE_EXTENSION, WARC_HEADER_ENCODING, WARC_ID, WARC_MAGIC, WARC_VERSION, WSP

Fields inherited from interface org.archive.format.ArchiveFileConstants

ABSOLUTE_OFFSET_KEY, CDX, CDX_FILE, CDX_LINE_BUFFER_SIZE, CRLF, DATE_FIELD_KEY, DEFAULT_DIGEST_METHOD, DOT_COMPRESSED_FILE_EXTENSION, DUMP, GZIP_DUMP, HEADER, INVALID_SUFFIX, LENGTH_FIELD_KEY, MIMETYPE_FIELD_KEY, NOHEAD, OCCUPIED_SUFFIX, ORIGIN_FIELD_KEY, READER_IDENTIFIER_FIELD_KEY, RECORD_IDENTIFIER_FIELD_KEY, SINGLE_SPACE, TYPE_FIELD_KEY, URL_FIELD_KEY, VERSION_FIELD_KEY

Constructor Summary

Constructors

Constructor and Description
WARCWriter(java.util.concurrent.atomic.AtomicInteger serialNo, java.io.OutputStream out, java.io.File f, WARCWriterPoolSettings settings) Constructor.
WARCWriter(java.util.concurrent.atomic.AtomicInteger serialNo, WARCWriterPoolSettings settings) Constructor.

Method Summary

Modifier and Type	Method and Description
protected void	baseCharacterCheck(char c, java.lang.String parameter)
protected java.lang.String	checkHeaderLineMimetypeParameter(java.lang.String parameter)
protected java.lang.String	checkHeaderValue(java.lang.String value)
protected java.lang.String	createFile(java.io.File file)
protected java.lang.String	createRecordHeader(WARCRecordInfo metaRecord)
protected java.net.URI	generateRecordId(java.util.Map<java.lang.String,java.lang.String> qualifiers)
protected java.net.URI	generateRecordId(java.lang.String key, java.lang.String value)
java.lang.String	getFilenameWithoutOccupiedSuffix()
static long	getStat(java.util.concurrent.ConcurrentMap<java.lang.String,java.util.concurrent.ConcurrentMap<java.lang.String,java.util.concurrent.atomic.AtomicLong>> map, java.lang.String key, java.lang.String subkey)
static long	getStat(java.util.Map<java.lang.String,java.util.Map<java.lang.String,java.lang.Long>> map, java.lang.String key, java.lang.String subkey)
java.lang.Iterable<WARCRecordInfo>	getTmpRecordLog()
java.util.Map<java.lang.String,java.util.Map<java.lang.String,java.lang.Long>>	getTmpStats()
void	resetTmpRecordLog()
void	resetTmpStats()
protected void	subtally(java.util.Map<java.lang.String,java.lang.Long> substats, long contentBytes, long totalBytes, long sizeOnDisk)
protected void	tally(WARCConstants.WARCRecordType warcRecordType, long contentBytes, long totalBytes, long sizeOnDisk)
void	writeRecord(WARCRecordInfo recordInfo)
java.net.URI	writeWarcinfoRecord(java.lang.String filename)
java.net.URI	writeWarcinfoRecord(java.lang.String filename, java.lang.String description)

Methods inherited from class org.archive.io.WriterPoolMember

checkSize, checkWriteable, close, copyFrom, createFile, flush, generateNewBasename, getBaseFilename, getFile, getNextDirectory, getOutputStream, getPosition, isCompressed, isOversize, postWriteRecordTasks, preWriteRecordTasks, write, write, write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TOTALS

public static final java.lang.String TOTALS

See Also:

Constant Field Values

SIZE_ON_DISK

public static final java.lang.String SIZE_ON_DISK

See Also:

Constant Field Values

TOTAL_BYTES

public static final java.lang.String TOTAL_BYTES

See Also:

Constant Field Values

CONTENT_BYTES

public static final java.lang.String CONTENT_BYTES

See Also:

Constant Field Values

NUM_RECORDS

public static final java.lang.String NUM_RECORDS

See Also:

Constant Field Values

CRLF_BYTES

public static byte[] CRLF_BYTES

NEWLINE as bytes.

Constructor Detail

WARCWriter

public WARCWriter(java.util.concurrent.atomic.AtomicInteger serialNo,
                  java.io.OutputStream out,
                  java.io.File f,
                  WARCWriterPoolSettings settings)
           throws java.io.IOException

Constructor. Takes a stream. Use with caution. There is no upperbound check on size. Will just keep writing. Only pass Streams that are bounded.

Parameters:

serialNo - used to generate unique file name sequences

out - Where to write.

f - File the out is connected to.

cmprs - Compress the content written.

a14DigitDate - If null, we'll write current time.

Throws:

java.io.IOException

WARCWriter

public WARCWriter(java.util.concurrent.atomic.AtomicInteger serialNo,
                  WARCWriterPoolSettings settings)

Constructor.

Parameters:

dirs - Where to drop files.

prefix - File prefix to use.

cmprs - Compress the records written.

maxSize - Maximum size for ARC files written.

suffix - File tail to use. If null, unused.

warcinfoData - File metadata for warcinfo record.

Method Detail

createFile

protected java.lang.String createFile(java.io.File file)
                               throws java.io.IOException

Overrides:

createFile in class WriterPoolMember

Throws:

java.io.IOException

baseCharacterCheck

protected void baseCharacterCheck(char c,
                                  java.lang.String parameter)
                           throws java.lang.IllegalArgumentException

Throws:

java.lang.IllegalArgumentException

checkHeaderValue

protected java.lang.String checkHeaderValue(java.lang.String value)
                                     throws java.lang.IllegalArgumentException

Throws:

java.lang.IllegalArgumentException

checkHeaderLineMimetypeParameter

protected java.lang.String checkHeaderLineMimetypeParameter(java.lang.String parameter)
                                                     throws java.lang.IllegalArgumentException

Throws:

java.lang.IllegalArgumentException

createRecordHeader

protected java.lang.String createRecordHeader(WARCRecordInfo metaRecord)
                                       throws java.lang.IllegalArgumentException

Throws:

java.lang.IllegalArgumentException

writeRecord

public void writeRecord(WARCRecordInfo recordInfo)
                 throws java.io.IOException

Throws:

java.io.IOException

getFilenameWithoutOccupiedSuffix

public java.lang.String getFilenameWithoutOccupiedSuffix()

tally

protected void tally(WARCConstants.WARCRecordType warcRecordType,
                     long contentBytes,
                     long totalBytes,
                     long sizeOnDisk)

subtally

protected void subtally(java.util.Map<java.lang.String,java.lang.Long> substats,
                        long contentBytes,
                        long totalBytes,
                        long sizeOnDisk)

generateRecordId

protected java.net.URI generateRecordId(java.util.Map<java.lang.String,java.lang.String> qualifiers)
                                 throws java.io.IOException

Throws:

java.io.IOException

generateRecordId

protected java.net.URI generateRecordId(java.lang.String key,
                                        java.lang.String value)
                                 throws java.io.IOException

Throws:

java.io.IOException

writeWarcinfoRecord

public java.net.URI writeWarcinfoRecord(java.lang.String filename)
                                 throws java.io.IOException

Throws:

java.io.IOException

writeWarcinfoRecord

public java.net.URI writeWarcinfoRecord(java.lang.String filename,
                                        java.lang.String description)
                                 throws java.io.IOException

Throws:

java.io.IOException

resetTmpStats

public void resetTmpStats()

See Also:

for usage model

getTmpStats

public java.util.Map<java.lang.String,java.util.Map<java.lang.String,java.lang.Long>> getTmpStats()

getStat

public static long getStat(java.util.Map<java.lang.String,java.util.Map<java.lang.String,java.lang.Long>> map,
                           java.lang.String key,
                           java.lang.String subkey)

getStat

public static long getStat(java.util.concurrent.ConcurrentMap<java.lang.String,java.util.concurrent.ConcurrentMap<java.lang.String,java.util.concurrent.atomic.AtomicLong>> map,
                           java.lang.String key,
                           java.lang.String subkey)

resetTmpRecordLog

public void resetTmpRecordLog()

getTmpRecordLog

public java.lang.Iterable<WARCRecordInfo> getTmpRecordLog()