Class WARCWriter

java.lang.Object

org.archive.io.WriterPoolMember

org.archive.io.warc.WARCWriter

public class WARCWriter
extends WriterPoolMember

WARC implementation.

Assumption is that the caller is managing access to this WARCWriter ensuring only one thread accessing this WARC instance at any one time.

While being written, WARCs have a '.open' suffix appended.

Version:

$Revision: 4604 $ $Date: 2006-09-05 22:38:18 -0700 (Tue, 05 Sep 2006) $

Author:

stack

Field Summary

Fields

Modifier and Type	Field	Description
static final String	CONTENT_BYTES
static byte[]	CRLF_BYTES	NEWLINE as bytes.
static final String	NUM_RECORDS
static final String	SIZE_ON_DISK
static final String	TOTAL_BYTES
static final String	TOTALS

Fields inherited from class org.archive.io.WriterPoolMember

countOut, currentBasename, currentTimestamp, DEFAULT_PREFIX, DEFAULT_TEMPLATE, f, out, roundRobinIndex, scratchbuffer, serialNoFormatter, serialNoFormatterSymbols, settings, UTF8

Constructor Summary

Constructors

Constructor	Description
WARCWriter(AtomicInteger serialNo, OutputStream out, File f, WARCWriterPoolSettings settings)	Constructor.
WARCWriter(AtomicInteger serialNo, WARCWriterPoolSettings settings)	Constructor.

Method Summary

Modifier and Type	Method	Description
protected void	baseCharacterCheck(char c, String parameter)
protected String	checkHeaderLineMimetypeParameter(String parameter)
protected String	checkHeaderValue(String value)
protected String	createFile(File file)
protected String	createRecordHeader(WARCRecordInfo metaRecord)
protected URI	generateRecordId(String key, String value)
protected URI	generateRecordId(Map<String,String> qualifiers)
String	getFilenameWithoutOccupiedSuffix()
static long	getStat(ConcurrentMap<String, ConcurrentMap<String, AtomicLong>> map, String key, String subkey)
static long	getStat(Map<String, Map<String,Long>> map, String key, String subkey)
Iterable<WARCRecordInfo>	getTmpRecordLog()
Map<String, Map<String,Long>>	getTmpStats()
void	resetTmpRecordLog()
void	resetTmpStats()
protected void	subtally(Map<String,Long> substats, long contentBytes, long totalBytes, long sizeOnDisk)
protected void	tally(WARCConstants.WARCRecordType warcRecordType, long contentBytes, long totalBytes, long sizeOnDisk)
void	writeRecord(WARCRecordInfo recordInfo)
URI	writeWarcinfoRecord(String filename)
URI	writeWarcinfoRecord(String filename, String description)

Methods inherited from class org.archive.io.WriterPoolMember

checkSize, checkWriteable, close, copyFrom, createFile, flush, generateNewBasename, getBaseFilename, getFile, getNextDirectory, getOutputStream, getPosition, isCompressed, isOversize, postWriteRecordTasks, preWriteRecordTasks, write, write, write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

TOTALS

public static final String TOTALS

See Also:

• Constant Field Values

SIZE_ON_DISK

public static final String SIZE_ON_DISK

See Also:

• Constant Field Values

TOTAL_BYTES

public static final String TOTAL_BYTES

See Also:

• Constant Field Values

CONTENT_BYTES

public static final String CONTENT_BYTES

See Also:

• Constant Field Values

NUM_RECORDS

public static final String NUM_RECORDS

See Also:

• Constant Field Values

CRLF_BYTES

public static byte[] CRLF_BYTES

NEWLINE as bytes.

Constructor Details

WARCWriter

public WARCWriter(AtomicInteger serialNo,
 OutputStream out,
 File f,
 WARCWriterPoolSettings settings)
           throws IOException

Constructor. Takes a stream. Use with caution. There is no upperbound check on size. Will just keep writing. Only pass Streams that are bounded.

Parameters:

serialNo - used to generate unique file name sequences

out - Where to write.

f - File the out is connected to.

Throws:

IOException

WARCWriter

public WARCWriter(AtomicInteger serialNo,
 WARCWriterPoolSettings settings)

Constructor.

Method Details

createFile

protected String createFile(File file)
                     throws IOException

Overrides:

createFile in class WriterPoolMember

Throws:

IOException

baseCharacterCheck

protected void baseCharacterCheck(char c,
 String parameter)
                           throws IllegalArgumentException

Throws:

IllegalArgumentException

checkHeaderValue

protected String checkHeaderValue(String value)
                           throws IllegalArgumentException

Throws:

IllegalArgumentException

checkHeaderLineMimetypeParameter

protected String checkHeaderLineMimetypeParameter(String parameter)
                                           throws IllegalArgumentException

Throws:

IllegalArgumentException

createRecordHeader

protected String createRecordHeader(WARCRecordInfo metaRecord)
                             throws IllegalArgumentException

Throws:

IllegalArgumentException

writeRecord

public void writeRecord(WARCRecordInfo recordInfo)
                 throws IOException

Throws:

IOException

getFilenameWithoutOccupiedSuffix

public String getFilenameWithoutOccupiedSuffix()

tally

protected void tally(WARCConstants.WARCRecordType warcRecordType,
 long contentBytes,
 long totalBytes,
 long sizeOnDisk)

subtally

protected void subtally(Map<String,Long> substats,
 long contentBytes,
 long totalBytes,
 long sizeOnDisk)

generateRecordId

protected URI generateRecordId(Map<String,String> qualifiers)
                        throws IOException

Throws:

IOException

generateRecordId

protected URI generateRecordId(String key,
 String value)
                        throws IOException

Throws:

IOException

writeWarcinfoRecord

public URI writeWarcinfoRecord(String filename)
                        throws IOException

Throws:

IOException

writeWarcinfoRecord

public URI writeWarcinfoRecord(String filename,
 String description)
                        throws IOException

Throws:

IOException

resetTmpStats

public void resetTmpStats()

See Also:

• for usage model

getTmpStats

public Map<String, Map<String,Long>> getTmpStats()

getStat

public static long getStat(Map<String, Map<String,Long>> map,
 String key,
 String subkey)

getStat

public static long getStat(ConcurrentMap<String, ConcurrentMap<String, AtomicLong>> map,
 String key,
 String subkey)

resetTmpRecordLog

public void resetTmpRecordLog()

getTmpRecordLog

public Iterable<WARCRecordInfo> getTmpRecordLog()