org.archive.io.warc

Class WARCWriter

java.lang.Object

org.archive.io.WriterPoolMember

org.archive.io.warc.WARCWriter

All Implemented Interfaces:

ArchiveFileConstants, WARCConstants

public class WARCWriter
extends WriterPoolMember
implements WARCConstants

WARC implementation.

Assumption is that the caller is managing access to this WARCWriter ensuring only one thread accessing this WARC instance at any one time.

While being written, WARCs have a '.open' suffix appended.

Version:

$Revision: 4604 $ $Date: 2006-09-05 22:38:18 -0700 (Tue, 05 Sep 2006) $

Nested Class Summary

Nested classes/interfaces inherited from interface org.archive.format.warc.WARCConstants

WARCConstants.WARCRecordType

Field Summary

Fields

Modifier and Type	Field and Description
static String	CONTENT_BYTES
static byte[]	CRLF_BYTES NEWLINE as bytes.
static String	NUM_RECORDS
static String	SIZE_ON_DISK
static String	TOTAL_BYTES
static String	TOTALS

Fields inherited from class org.archive.io.WriterPoolMember

countOut, currentBasename, currentTimestamp, DEFAULT_PREFIX, DEFAULT_TEMPLATE, f, out, roundRobinIndex, scratchbuffer, serialNoFormatter, settings, UTF8

Fields inherited from interface org.archive.format.warc.WARCConstants

COLON_SPACE, COMPRESSED_WARC_FILE_EXTENSION, CONTENT_DESCRIPTION, CONTENT_LENGTH, CONTENT_TYPE, DEFAULT_ENCODING, DEFAULT_MAX_WARC_FILE_SIZE, DOT_COMPRESSED_WARC_FILE_EXTENSION, DOT_WARC_FILE_EXTENSION, FTP_CONTROL_CONVERSATION_MIMETYPE, HEADER_FIELD_SEPARATOR, HEADER_KEY_BLOCK_DIGEST, HEADER_KEY_CONCURRENT_TO, HEADER_KEY_DATE, HEADER_KEY_ETAG, HEADER_KEY_FILENAME, HEADER_KEY_ID, HEADER_KEY_IP, HEADER_KEY_LAST_MODIFIED, HEADER_KEY_PAYLOAD_DIGEST, HEADER_KEY_PROFILE, HEADER_KEY_REFERS_TO, HEADER_KEY_REFERS_TO_DATE, HEADER_KEY_REFERS_TO_FILE_OFFSET, HEADER_KEY_REFERS_TO_FILENAME, HEADER_KEY_REFERS_TO_TARGET_URI, HEADER_KEY_TRUNCATED, HEADER_KEY_TYPE, HEADER_KEY_URI, HEADER_LINE_ENCODING, HTTP_REQUEST_MIMETYPE, HTTP_RESPONSE_MIMETYPE, MAX_LINE_LENGTH, MAX_WARC_HEADER_LINE_LENGTH, NAMED_FIELD_CHECKSUM_LABEL, NAMED_FIELD_DESCRIPTION, NAMED_FIELD_FILEDESC, NAMED_FIELD_IP_LABEL, NAMED_FIELD_RELATED_LABEL, NAMED_FIELD_TRUNCATED, NAMED_FIELD_TRUNCATED_VALUE_HEAD, NAMED_FIELD_TRUNCATED_VALUE_LENGTH, NAMED_FIELD_TRUNCATED_VALUE_TIME, NAMED_FIELD_TRUNCATED_VALUE_UNSPECIFIED, NAMED_FIELD_WARCFILENAME, PLACEHOLDER_RECORD_LENGTH_STRING, PROFILE_REVISIT_IDENTICAL_DIGEST, PROFILE_REVISIT_NOT_MODIFIED, PROFILE_REVISIT_URI_AGNOSTIC_IDENTICAL_DIGEST, TRUNCATED_VALUE_UNSPECIFIED, TYPE, WARC_FIELDS_TYPE, WARC_FILE_EXTENSION, WARC_HEADER_ENCODING, WARC_ID, WARC_MAGIC, WARC_VERSION, WSP

Fields inherited from interface org.archive.format.ArchiveFileConstants

ABSOLUTE_OFFSET_KEY, CDX, CDX_FILE, CDX_LINE_BUFFER_SIZE, CRLF, DATE_FIELD_KEY, DEFAULT_DIGEST_METHOD, DOT_COMPRESSED_FILE_EXTENSION, DUMP, GZIP_DUMP, HEADER, INVALID_SUFFIX, LENGTH_FIELD_KEY, MIMETYPE_FIELD_KEY, NOHEAD, OCCUPIED_SUFFIX, READER_IDENTIFIER_FIELD_KEY, RECORD_IDENTIFIER_FIELD_KEY, SINGLE_SPACE, TYPE_FIELD_KEY, URL_FIELD_KEY, VERSION_FIELD_KEY

Constructor Summary

Constructors

Constructor and Description
WARCWriter(AtomicInteger serialNo, OutputStream out, File f, WARCWriterPoolSettings settings) Constructor.
WARCWriter(AtomicInteger serialNo, WARCWriterPoolSettings settings) Constructor.

Method Summary

Methods

Modifier and Type	Method and Description
protected void	baseCharacterCheck(char c, String parameter)
protected String	checkHeaderLineMimetypeParameter(String parameter)
protected String	checkHeaderValue(String value)
protected String	createFile(File file)
protected String	createRecordHeader(WARCRecordInfo metaRecord)
protected URI	generateRecordId(Map<String,String> qualifiers)
protected URI	generateRecordId(String key, String value)
String	getFilenameWithoutOccupiedSuffix()
static long	getStat(ConcurrentMap<String,ConcurrentMap<String,AtomicLong>> map, String key, String subkey)
static long	getStat(Map<String,Map<String,Long>> map, String key, String subkey)
Iterable<WARCRecordInfo>	getTmpRecordLog()
Map<String,Map<String,Long>>	getTmpStats()
void	resetTmpRecordLog()
void	resetTmpStats()
protected void	subtally(Map<String,Long> substats, long contentBytes, long totalBytes, long sizeOnDisk)
protected void	tally(WARCConstants.WARCRecordType warcRecordType, long contentBytes, long totalBytes, long sizeOnDisk)
void	writeRecord(WARCRecordInfo recordInfo)
URI	writeWarcinfoRecord(String filename)
URI	writeWarcinfoRecord(String filename, String description)

Methods inherited from class org.archive.io.WriterPoolMember

checkSize, checkWriteable, close, copyFrom, createFile, flush, generateNewBasename, getBaseFilename, getFile, getNextDirectory, getOutputStream, getPosition, isCompressed, isOversize, postWriteRecordTasks, preWriteRecordTasks, write, write, write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TOTALS

public static final String TOTALS

See Also:

Constant Field Values

SIZE_ON_DISK

public static final String SIZE_ON_DISK

See Also:

Constant Field Values

TOTAL_BYTES

public static final String TOTAL_BYTES

See Also:

Constant Field Values

CONTENT_BYTES

public static final String CONTENT_BYTES

See Also:

Constant Field Values

NUM_RECORDS

public static final String NUM_RECORDS

See Also:

Constant Field Values

CRLF_BYTES

public static byte[] CRLF_BYTES

NEWLINE as bytes.

Constructor Detail

WARCWriter

public WARCWriter(AtomicInteger serialNo,
          OutputStream out,
          File f,
          WARCWriterPoolSettings settings)
           throws IOException

Constructor. Takes a stream. Use with caution. There is no upperbound check on size. Will just keep writing. Only pass Streams that are bounded.

Parameters:

serialNo - used to generate unique file name sequences

out - Where to write.

f - File the out is connected to.

cmprs - Compress the content written.

a14DigitDate - If null, we'll write current time.

Throws:

IOException

WARCWriter

public WARCWriter(AtomicInteger serialNo,
          WARCWriterPoolSettings settings)

Constructor.

Parameters:

dirs - Where to drop files.

prefix - File prefix to use.

cmprs - Compress the records written.

maxSize - Maximum size for ARC files written.

suffix - File tail to use. If null, unused.

warcinfoData - File metadata for warcinfo record.

Method Detail

createFile

protected String createFile(File file)
                     throws IOException

Overrides:

createFile in class WriterPoolMember

Throws:

IOException

baseCharacterCheck

protected void baseCharacterCheck(char c,
                      String parameter)
                           throws IllegalArgumentException

Throws:

IllegalArgumentException

checkHeaderValue

protected String checkHeaderValue(String value)
                           throws IllegalArgumentException

Throws:

IllegalArgumentException

checkHeaderLineMimetypeParameter

protected String checkHeaderLineMimetypeParameter(String parameter)
                                           throws IllegalArgumentException

Throws:

IllegalArgumentException

createRecordHeader

protected String createRecordHeader(WARCRecordInfo metaRecord)
                             throws IllegalArgumentException

Throws:

IllegalArgumentException

writeRecord

public void writeRecord(WARCRecordInfo recordInfo)
                 throws IOException

Throws:

IOException

getFilenameWithoutOccupiedSuffix

public String getFilenameWithoutOccupiedSuffix()

tally

protected void tally(WARCConstants.WARCRecordType warcRecordType,
         long contentBytes,
         long totalBytes,
         long sizeOnDisk)

subtally

protected void subtally(Map<String,Long> substats,
            long contentBytes,
            long totalBytes,
            long sizeOnDisk)

generateRecordId

protected URI generateRecordId(Map<String,String> qualifiers)
                        throws IOException

Throws:

IOException

generateRecordId

protected URI generateRecordId(String key,
                   String value)
                        throws IOException

Throws:

IOException

writeWarcinfoRecord

public URI writeWarcinfoRecord(String filename)
                        throws IOException

Throws:

IOException

writeWarcinfoRecord

public URI writeWarcinfoRecord(String filename,
                      String description)
                        throws IOException

Throws:

IOException

resetTmpStats

public void resetTmpStats()

See Also:

for usage model

getTmpStats

public Map<String,Map<String,Long>> getTmpStats()

getStat

public static long getStat(Map<String,Map<String,Long>> map,
           String key,
           String subkey)

getStat

public static long getStat(ConcurrentMap<String,ConcurrentMap<String,AtomicLong>> map,
           String key,
           String subkey)

resetTmpRecordLog

public void resetTmpRecordLog()

getTmpRecordLog

public Iterable<WARCRecordInfo> getTmpRecordLog()