org.archive.io

Class WriterPoolMember

java.lang.Object

org.archive.io.WriterPoolMember

All Implemented Interfaces:

ArchiveFileConstants

Direct Known Subclasses:

ARCWriter, WARCWriter

public abstract class WriterPoolMember
extends Object
implements ArchiveFileConstants

Member of WriterPool. Implements rotating off files, file naming with some guarantee of uniqueness, and position in file. Subclass to pick up functionality for a particular Writer type.

Version:

$Date$ $Revision$

Author:

stack

Field Summary

Fields

Modifier and Type	Field and Description
protected MiserOutputStream	countOut Counting stream for metering
protected String	currentBasename
protected String	currentTimestamp Creation date for the current file.
static String	DEFAULT_PREFIX Default for file prefix.
static String	DEFAULT_TEMPLATE Default archival-aggregate filename template.
protected File	f Reference to file we're currently writing.
protected OutputStream	out Output stream for file.
protected static int	roundRobinIndex Directories round-robin index.
protected byte[]	scratchbuffer Buffer to reuse writing streams.
protected static NumberFormat	serialNoFormatter NumberFormat instance for formatting serial number.
protected WriterPoolSettings	settings
static String	UTF8

Fields inherited from interface org.archive.format.ArchiveFileConstants

ABSOLUTE_OFFSET_KEY, CDX, CDX_FILE, CDX_LINE_BUFFER_SIZE, CRLF, DATE_FIELD_KEY, DEFAULT_DIGEST_METHOD, DOT_COMPRESSED_FILE_EXTENSION, DUMP, GZIP_DUMP, HEADER, INVALID_SUFFIX, LENGTH_FIELD_KEY, MIMETYPE_FIELD_KEY, NOHEAD, OCCUPIED_SUFFIX, READER_IDENTIFIER_FIELD_KEY, RECORD_IDENTIFIER_FIELD_KEY, SINGLE_SPACE, TYPE_FIELD_KEY, URL_FIELD_KEY, VERSION_FIELD_KEY

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	WriterPoolMember(AtomicInteger serialNo, OutputStream out, File file, WriterPoolSettings settings) Constructor.
	WriterPoolMember(AtomicInteger serialNo, WriterPoolSettings settings, String extension) Constructor.

Method Summary

Methods

Modifier and Type	Method and Description
void	checkSize() Call this method just before/after any significant write.
protected File	checkWriteable(File d)
void	close()
protected long	copyFrom(InputStream is, long recordLength, boolean enforceLength) Copy bytes from the provided InputStream to the target file/stream being written.
protected String	createFile() Create a new file.
protected String	createFile(File file)
protected void	flush()
protected void	generateNewBasename() Generate a new basename by interpolating values in the configured template.
protected String	getBaseFilename() Get the file name
File	getFile() Get this file.
protected File	getNextDirectory(List<File> dirs)
protected OutputStream	getOutputStream()
long	getPosition() Position in raw output (typically, physical file).
boolean	isCompressed()
boolean	isOversize() Check if underlying file has already reached its target size.
protected void	postWriteRecordTasks() Post file write tasks.
protected void	preWriteRecordTasks() Post write tasks.
protected void	write(byte[] b)
protected void	write(byte[] b, int off, int len)
protected void	write(int b)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

UTF8

public static final String UTF8

See Also:

Constant Field Values

DEFAULT_TEMPLATE

public static final String DEFAULT_TEMPLATE

Default archival-aggregate filename template. Under usual assumptions -- hostnames aren't shared among crawling hosts; processes have unique PIDs and admin ports; timestamps inside one process don't repeat (see UniqueTimestampService); clocks are generally accurate -- will generate a unique name. Stands for Internet Archive Heritrix.

See Also:

Constant Field Values

DEFAULT_PREFIX

public static final String DEFAULT_PREFIX

Default for file prefix.

See Also:

Constant Field Values

f

protected File f

Reference to file we're currently writing.

out

protected OutputStream out

Output stream for file.

countOut

protected MiserOutputStream countOut

Counting stream for metering

settings

protected WriterPoolSettings settings

currentTimestamp

protected String currentTimestamp

Creation date for the current file. Set by createFile().

currentBasename

protected String currentBasename

roundRobinIndex

protected static int roundRobinIndex

Directories round-robin index.

serialNoFormatter

protected static NumberFormat serialNoFormatter

NumberFormat instance for formatting serial number. Pads serial number with zeros.

scratchbuffer

protected final byte[] scratchbuffer

Buffer to reuse writing streams.

Constructor Detail

WriterPoolMember

protected WriterPoolMember(AtomicInteger serialNo,
                OutputStream out,
                File file,
                WriterPoolSettings settings)
                    throws IOException

Constructor. Takes a stream. Use with caution. There is no upperbound check on size. Will just keep writing.

Parameters:

serialNo - used to create unique filename sequences

out - Where to write.

file - File the out is connected to.

cmprs - Compress the content written.

a14DigitDate - If null, we'll write current time.

Throws:

IOException

WriterPoolMember

public WriterPoolMember(AtomicInteger serialNo,
                WriterPoolSettings settings,
                String extension)

Constructor.

Parameters:

serialNo - used to create unique filename sequences

dirs - Where to drop files.

prefix - File prefix to use.

cmprs - Compress the records written.

maxSize - Maximum size for ARC files written.

template - filenaming template to use

extension - Extension to give file.

Method Detail

checkSize

public void checkSize()
               throws IOException

Call this method just before/after any significant write. Call at the end of the writing of a record or just before we start writing a new record. Will close current file and open a new file if file size has passed out maxSize.

Creates and opens a file if none already open. One use of this method then is after construction, call this method to add the metadata, then call getPosition() to find offset of first record. TODO: perhaps this should be called checkForNewOpen? because it also handles initial open, even when not rolling oversize

Throws:

IOException

isOversize

public boolean isOversize()

Check if underlying file has already reached its target size.

Returns:

boolean true if file has reached target size and due to be closed

createFile

protected String createFile()
                     throws IOException

Create a new file. Rotates off the current Writer and creates a new in its place to take subsequent writes. Usually called from checkSize().

Returns:

Name of file created.

Throws:

IOException

createFile

protected String createFile(File file)
                     throws IOException

Throws:

IOException

getNextDirectory

protected File getNextDirectory(List<File> dirs)
                         throws IOException

Parameters:

dirs - List of File objects that point at directories.

Returns:

Find next directory to write an arc too. If more than one, it tries to round-robin through each in turn.

Throws:

IOException

checkWriteable

protected File checkWriteable(File d)

generateNewBasename

protected void generateNewBasename()

Generate a new basename by interpolating values in the configured template. Values come from local state, other configured values, and global system properties. The recommended default template will generate a unique basename under reasonable assumptions.

getBaseFilename

protected String getBaseFilename()

Get the file name

Returns:

the filename, as if uncompressed

getFile

public File getFile()

Get this file. Used by junit test to test for creation and when WriterPool wants to invalidate a file.

Returns:

The current file.

preWriteRecordTasks

protected void preWriteRecordTasks()
                            throws IOException

Post write tasks. Has side effects. Will open new file if we're at the upper bound. If we're writing compressed files, it will wrap output stream with a GZIP writer with side effect that GZIP header is written out on the stream.

Throws:

IOException

postWriteRecordTasks

protected void postWriteRecordTasks()
                             throws IOException

Post file write tasks. If compressed, finishes up compression and flushes stream so any subsequent checks get good reading.

Throws:

IOException

getPosition

public long getPosition()

Position in raw output (typically, physical file). Used making accounting of bytes written.

Returns:

Position in final media (assuming all flushing completes)

Throws:

IOException

isCompressed

public boolean isCompressed()

write

protected void write(byte[] b)
              throws IOException

Throws:

IOException

flush

protected void flush()
              throws IOException

Throws:

IOException

write

protected void write(byte[] b,
         int off,
         int len)
              throws IOException

Throws:

IOException

write

protected void write(int b)
              throws IOException

Throws:

IOException

copyFrom

protected long copyFrom(InputStream is,
            long recordLength,
            boolean enforceLength)
                 throws IOException

Copy bytes from the provided InputStream to the target file/stream being written.

Parameters:

is - InputStream to copy bytes from

recordLength - expected number of bytes to copy

enforceLength - whether to throw an exception if too many/too few bytes are available from stream

Returns:

number of bytes written (normally equal to enforceLength)

Throws:

IOException

close

public void close()
           throws IOException

Throws:

IOException

getOutputStream

protected OutputStream getOutputStream()