org.archive.io

Class WriterPoolMember

java.lang.Object

org.archive.io.WriterPoolMember

All Implemented Interfaces:

ArchiveFileConstants

Direct Known Subclasses:

ARCWriter, WARCWriter

public abstract class WriterPoolMember
extends java.lang.Object
implements ArchiveFileConstants

Member of WriterPool. Implements rotating off files, file naming with some guarantee of uniqueness, and position in file. Subclass to pick up functionality for a particular Writer type.

Version:

$Date$ $Revision$

Author:

stack

Field Summary

Fields

Modifier and Type	Field and Description
protected MiserOutputStream	countOut Counting stream for metering
protected java.lang.String	currentBasename
protected java.lang.String	currentTimestamp Creation date for the current file.
static java.lang.String	DEFAULT_PREFIX Default for file prefix.
static java.lang.String	DEFAULT_TEMPLATE Default archival-aggregate filename template.
protected java.io.File	f Reference to file we're currently writing.
protected java.io.OutputStream	out Output stream for file.
protected static int	roundRobinIndex Directories round-robin index.
protected byte[]	scratchbuffer Buffer to reuse writing streams.
protected static java.text.NumberFormat	serialNoFormatter NumberFormat instance for formatting serial number.
protected WriterPoolSettings	settings
static java.lang.String	UTF8

Fields inherited from interface org.archive.format.ArchiveFileConstants

ABSOLUTE_OFFSET_KEY, CDX, CDX_FILE, CDX_LINE_BUFFER_SIZE, CRLF, DATE_FIELD_KEY, DEFAULT_DIGEST_METHOD, DOT_COMPRESSED_FILE_EXTENSION, DUMP, GZIP_DUMP, HEADER, INVALID_SUFFIX, LENGTH_FIELD_KEY, MIMETYPE_FIELD_KEY, NOHEAD, OCCUPIED_SUFFIX, ORIGIN_FIELD_KEY, READER_IDENTIFIER_FIELD_KEY, RECORD_IDENTIFIER_FIELD_KEY, SINGLE_SPACE, TYPE_FIELD_KEY, URL_FIELD_KEY, VERSION_FIELD_KEY

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	WriterPoolMember(java.util.concurrent.atomic.AtomicInteger serialNo, java.io.OutputStream out, java.io.File file, WriterPoolSettings settings) Constructor.
	WriterPoolMember(java.util.concurrent.atomic.AtomicInteger serialNo, WriterPoolSettings settings, java.lang.String extension) Constructor.

Method Summary

Modifier and Type	Method and Description
void	checkSize() Call this method just before/after any significant write.
protected java.io.File	checkWriteable(java.io.File d)
void	close()
protected long	copyFrom(java.io.InputStream is, long recordLength, boolean enforceLength) Copy bytes from the provided InputStream to the target file/stream being written.
protected java.lang.String	createFile() Create a new file.
protected java.lang.String	createFile(java.io.File file)
protected void	flush()
protected void	generateNewBasename() Generate a new basename by interpolating values in the configured template.
protected java.lang.String	getBaseFilename() Get the file name
java.io.File	getFile() Get this file.
protected java.io.File	getNextDirectory(java.util.List<java.io.File> dirs)
protected java.io.OutputStream	getOutputStream()
long	getPosition() Position in raw output (typically, physical file).
boolean	isCompressed()
boolean	isOversize() Check if underlying file has already reached its target size.
protected void	postWriteRecordTasks() Post file write tasks.
protected void	preWriteRecordTasks() Post write tasks.
protected void	write(byte[] b)
protected void	write(byte[] b, int off, int len)
protected void	write(int b)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

UTF8

public static final java.lang.String UTF8

See Also:

Constant Field Values

DEFAULT_TEMPLATE

public static final java.lang.String DEFAULT_TEMPLATE

Default archival-aggregate filename template. Under usual assumptions -- hostnames aren't shared among crawling hosts; processes have unique PIDs and admin ports; timestamps inside one process don't repeat (see UniqueTimestampService); clocks are generally accurate -- will generate a unique name. Stands for Internet Archive Heritrix.

See Also:

Constant Field Values

DEFAULT_PREFIX

public static final java.lang.String DEFAULT_PREFIX

Default for file prefix.

See Also:

Constant Field Values

f

protected java.io.File f

Reference to file we're currently writing.

out

protected java.io.OutputStream out

Output stream for file.

countOut

protected MiserOutputStream countOut

Counting stream for metering

settings

protected WriterPoolSettings settings

currentTimestamp

protected java.lang.String currentTimestamp

Creation date for the current file. Set by createFile().

currentBasename

protected java.lang.String currentBasename

roundRobinIndex

protected static int roundRobinIndex

Directories round-robin index.

serialNoFormatter

protected static java.text.NumberFormat serialNoFormatter

NumberFormat instance for formatting serial number. Pads serial number with zeros.

scratchbuffer

protected final byte[] scratchbuffer

Buffer to reuse writing streams.

Constructor Detail

WriterPoolMember

protected WriterPoolMember(java.util.concurrent.atomic.AtomicInteger serialNo,
                           java.io.OutputStream out,
                           java.io.File file,
                           WriterPoolSettings settings)
                    throws java.io.IOException

Constructor. Takes a stream. Use with caution. There is no upperbound check on size. Will just keep writing.

Parameters:

serialNo - used to create unique filename sequences

out - Where to write.

file - File the out is connected to.

cmprs - Compress the content written.

a14DigitDate - If null, we'll write current time.

Throws:

java.io.IOException

WriterPoolMember

public WriterPoolMember(java.util.concurrent.atomic.AtomicInteger serialNo,
                        WriterPoolSettings settings,
                        java.lang.String extension)

Constructor.

Parameters:

serialNo - used to create unique filename sequences

dirs - Where to drop files.

prefix - File prefix to use.

cmprs - Compress the records written.

maxSize - Maximum size for ARC files written.

template - filenaming template to use

extension - Extension to give file.

Method Detail

checkSize

public void checkSize()
               throws java.io.IOException

Call this method just before/after any significant write. Call at the end of the writing of a record or just before we start writing a new record. Will close current file and open a new file if file size has passed out maxSize.

Creates and opens a file if none already open. One use of this method then is after construction, call this method to add the metadata, then call getPosition() to find offset of first record. TODO: perhaps this should be called checkForNewOpen? because it also handles initial open, even when not rolling oversize

Throws:

java.io.IOException

isOversize

public boolean isOversize()

Check if underlying file has already reached its target size.

Returns:

boolean true if file has reached target size and due to be closed

createFile

protected java.lang.String createFile()
                               throws java.io.IOException

Create a new file. Rotates off the current Writer and creates a new in its place to take subsequent writes. Usually called from checkSize().

Returns:

Name of file created.

Throws:

java.io.IOException

createFile

protected java.lang.String createFile(java.io.File file)
                               throws java.io.IOException

Throws:

java.io.IOException

getNextDirectory

protected java.io.File getNextDirectory(java.util.List<java.io.File> dirs)
                                 throws java.io.IOException

Parameters:

dirs - List of File objects that point at directories.

Returns:

Find next directory to write an arc too. If more than one, it tries to round-robin through each in turn.

Throws:

java.io.IOException

checkWriteable

protected java.io.File checkWriteable(java.io.File d)

generateNewBasename

protected void generateNewBasename()

Generate a new basename by interpolating values in the configured template. Values come from local state, other configured values, and global system properties. The recommended default template will generate a unique basename under reasonable assumptions.

getBaseFilename

protected java.lang.String getBaseFilename()

Get the file name

Returns:

the filename, as if uncompressed

getFile

public java.io.File getFile()

Get this file. Used by junit test to test for creation and when WriterPool wants to invalidate a file.

Returns:

The current file.

preWriteRecordTasks

protected void preWriteRecordTasks()
                            throws java.io.IOException

Post write tasks. Has side effects. Will open new file if we're at the upper bound. If we're writing compressed files, it will wrap output stream with a GZIP writer with side effect that GZIP header is written out on the stream.

Throws:

java.io.IOException

postWriteRecordTasks

protected void postWriteRecordTasks()
                             throws java.io.IOException

Post file write tasks. If compressed, finishes up compression and flushes stream so any subsequent checks get good reading.

Throws:

java.io.IOException

getPosition

public long getPosition()

Position in raw output (typically, physical file). Used making accounting of bytes written.

Returns:

Position in final media (assuming all flushing completes)

Throws:

java.io.IOException

isCompressed

public boolean isCompressed()

write

protected void write(byte[] b)
              throws java.io.IOException

Throws:

java.io.IOException

flush

protected void flush()
              throws java.io.IOException

Throws:

java.io.IOException

write

protected void write(byte[] b,
                     int off,
                     int len)
              throws java.io.IOException

Throws:

java.io.IOException

write

protected void write(int b)
              throws java.io.IOException

Throws:

java.io.IOException

copyFrom

protected long copyFrom(java.io.InputStream is,
                        long recordLength,
                        boolean enforceLength)
                 throws java.io.IOException

Copy bytes from the provided InputStream to the target file/stream being written.

Parameters:

is - InputStream to copy bytes from

recordLength - expected number of bytes to copy

enforceLength - whether to throw an exception if too many/too few bytes are available from stream

Returns:

number of bytes written (normally equal to enforceLength)

Throws:

java.io.IOException

close

public void close()
           throws java.io.IOException

Throws:

java.io.IOException

getOutputStream

protected java.io.OutputStream getOutputStream()