Class WriterPoolMember

java.lang.Object
org.archive.io.WriterPoolMember
Direct Known Subclasses:
ARCWriter, WARCWriter
public abstract class WriterPoolMember extends Object
Member of WriterPool. Implements rotating off files, file naming with some guarantee of uniqueness, and position in file. Subclass to pick up functionality for a particular Writer type.
Version:
$Date$ $Revision$
Author:
stack

  • Field Details

    • UTF8

      public static final String UTF8

    • DEFAULT_TEMPLATE

      public static final String DEFAULT_TEMPLATE
      Default archival-aggregate filename template. Under usual assumptions -- hostnames aren't shared among crawling hosts; processes have unique PIDs and admin ports; timestamps inside one process don't repeat (see UniqueTimestampService); clocks are generally accurate -- will generate a unique name. Stands for Internet Archive Heritrix.
      See Also:

    • DEFAULT_PREFIX

      public static final String DEFAULT_PREFIX
      Default for file prefix.
      See Also:

    • f

      protected File f
      Reference to file we're currently writing.

    • out

      protected OutputStream out
      Output stream for file.

    • countOut

      protected MiserOutputStream countOut
      Counting stream for metering

    • settings

      protected WriterPoolSettings settings

    • currentTimestamp

      protected String currentTimestamp
      Creation date for the current file. Set by createFile().

    • currentBasename

      protected String currentBasename

    • roundRobinIndex

      protected static int roundRobinIndex
      Directories round-robin index.

    • serialNoFormatterSymbols

      protected static DecimalFormatSymbols serialNoFormatterSymbols
      Symbol set for serial number formatter.

    • serialNoFormatter

      protected static NumberFormat serialNoFormatter
      NumberFormat instance for formatting serial number. Pads serial number with zeros.

    • scratchbuffer

      protected final byte[] scratchbuffer
      Buffer to reuse writing streams.

  • Constructor Details

    • WriterPoolMember

      protected WriterPoolMember(AtomicInteger serialNo, OutputStream out, File file, WriterPoolSettings settings) throws IOException
      Constructor. Takes a stream. Use with caution. There is no upperbound check on size. Will just keep writing.
      Parameters:
      serialNo - used to create unique filename sequences
      out - Where to write.
      file - File the out is connected to.
      Throws:
      IOException

    • WriterPoolMember

      public WriterPoolMember(AtomicInteger serialNo, WriterPoolSettings settings, String extension)
      Constructor.
      Parameters:
      serialNo - used to create unique filename sequences
      extension - Extension to give file.

  • Method Details

    • checkSize

      public void checkSize() throws IOException
      Call this method just before/after any significant write. Call at the end of the writing of a record or just before we start writing a new record. Will close current file and open a new file if file size has passed out maxSize.

      Creates and opens a file if none already open. One use of this method then is after construction, call this method to add the metadata, then call getPosition() to find offset of first record. TODO: perhaps this should be called checkForNewOpen? because it also handles initial open, even when not rolling oversize

      Throws:
      IOException

    • isOversize

      public boolean isOversize()
      Check if underlying file has already reached its target size.
      Returns:
      boolean true if file has reached target size and due to be closed

    • createFile

      protected String createFile() throws IOException
      Create a new file. Rotates off the current Writer and creates a new in its place to take subsequent writes. Usually called from checkSize().
      Returns:
      Name of file created.
      Throws:
      IOException

    • createFile

      protected String createFile(File file) throws IOException
      Throws:
      IOException

    • getNextDirectory

      protected File getNextDirectory(List<File> dirs) throws IOException
      Parameters:
      dirs - List of File objects that point at directories.
      Returns:
      Find next directory to write an arc too. If more than one, it tries to round-robin through each in turn.
      Throws:
      IOException

    • checkWriteable

      protected File checkWriteable(File d)

    • generateNewBasename

      protected void generateNewBasename()
      Generate a new basename by interpolating values in the configured template. Values come from local state, other configured values, and global system properties. The recommended default template will generate a unique basename under reasonable assumptions.

    • getBaseFilename

      protected String getBaseFilename()
      Get the file name
      Returns:
      the filename, as if uncompressed

    • getFile

      public File getFile()
      Get this file. Used by junit test to test for creation and when WriterPool wants to invalidate a file.
      Returns:
      The current file.

    • preWriteRecordTasks

      protected void preWriteRecordTasks() throws IOException
      Post write tasks. Has side effects. Will open new file if we're at the upper bound. If we're writing compressed files, it will wrap output stream with a GZIP writer with side effect that GZIP header is written out on the stream.
      Throws:
      IOException

    • postWriteRecordTasks

      protected void postWriteRecordTasks() throws IOException
      Post file write tasks. If compressed, finishes up compression and flushes stream so any subsequent checks get good reading.
      Throws:
      IOException

    • getPosition

      public long getPosition()
      Position in raw output (typically, physical file). Used making accounting of bytes written.
      Returns:
      Position in final media (assuming all flushing completes)

    • isCompressed

      public boolean isCompressed()

    • write

      protected void write(byte[] b) throws IOException
      Throws:
      IOException

    • flush

      protected void flush() throws IOException
      Throws:
      IOException

    • write

      protected void write(byte[] b, int off, int len) throws IOException
      Throws:
      IOException

    • write

      protected void write(int b) throws IOException
      Throws:
      IOException

    • copyFrom

      protected long copyFrom(InputStream is, long recordLength, boolean enforceLength) throws IOException
      Copy bytes from the provided InputStream to the target file/stream being written.
      Parameters:
      is - InputStream to copy bytes from
      recordLength - expected number of bytes to copy
      enforceLength - whether to throw an exception if too many/too few bytes are available from stream
      Returns:
      number of bytes written (normally equal to enforceLength)
      Throws:
      IOException

    • close

      public void close() throws IOException
      Throws:
      IOException

    • getOutputStream

      protected OutputStream getOutputStream()