Class RecordingOutputStream

java.lang.Object
java.io.OutputStream
org.archive.io.RecordingOutputStream
All Implemented Interfaces:
Closeable, Flushable, AutoCloseable
public class RecordingOutputStream extends OutputStream
An output stream that records all writes to wrapped output stream. A RecordingOutputStream can be wrapped around any other OutputStream to record all bytes written to it. You can then request a ReplayInputStream to read those bytes.

The RecordingOutputStream uses an in-memory buffer and backing disk file to allow it to record streams of arbitrary length limited only by available disk space.

As long as the stream recorded is smaller than the in-memory buffer, no disk access will occur.

Recorded content can be recovered as a ReplayInputStream (via getReplayInputStream() or, for only the content after the content-begin-mark is set, getContentReplayInputStream() ) or as a ReplayCharSequence (via getReplayCharSequence()).

This class is also used as a straight output stream by RecordingInputStream to which it records all reads. RecordingInputStream is exploiting the file backed buffer facility of this class passing null for the stream to wrap. TODO: Make a FileBackedOutputStream class that is subclassed by RecordingInputStream.

Author:
gojomo

  • Field Details

    • logger

      protected static Logger logger

    • size

      protected long size
      Size of recording. Later passed to ReplayInputStream on creation. It uses it to know when EOS.

    • backingFilename

      protected String backingFilename

    • diskStream

      protected OutputStream diskStream

    • MAX_HEADER_MATERIAL

      protected static final long MAX_HEADER_MATERIAL
      Maximum amount of header material to accept without the content body beginning -- if more, throw a RecorderTooMuchHeaderException. TODO: make configurable? make smaller?
      See Also:

    • maxLength

      protected long maxLength
      maximum length of material to record before throwing exception

    • timeoutMs

      protected long timeoutMs
      maximum time to record before throwing exception

    • maxRateBytesPerMs

      protected long maxRateBytesPerMs
      maximum rate to record (adds delays to hit target rate)

    • startTime

      protected long startTime
      time recording begins for timeout, rate calculations

    • messageBodyBeginMark

      protected long messageBodyBeginMark
      When recording HTTP, where the content-body starts.

    • lastTwoBytes

      protected int[] lastTwoBytes
      While messageBodyBeginMark is not set, the last two bytes seen.

      This class does automatic detection of http message body begin (i.e. end of http headers). Unfortunately httpcomponents did not want to add functionality to help us with this, see https://issues.apache.org/jira/browse/HTTPCORE-325

      It works like this: while messageBodyBeginMark is not set, we remember the last two bytes seen, and look at each byte we write. If the lastTwoBytes+currentByte is "\n\r\n", or lastTwoBytes[1]+currentByte is "\n\n" then we call markMessageBodyBegin() at the position after currentByte.

      An assumption here is that protocols other than http don't have headers, and for those protocols the user of this class will call markMessageBodyBegin() at position 0 before writing anything.

  • Constructor Details

    • RecordingOutputStream

      public RecordingOutputStream(int bufferSize, String backingFilename)
      Create a new RecordingOutputStream.
      Parameters:
      bufferSize - Buffer size to use.
      backingFilename - Name of backing file to use.

  • Method Details

    • open

      public void open() throws IOException
      Wrap the given stream, both recording and passing along any data written to this RecordingOutputStream.
      Throws:
      IOException - If failed creation of backing file.

    • open

      public void open(OutputStream wrappedStream) throws IOException
      Wrap the given stream, both recording and passing along any data written to this RecordingOutputStream.
      Parameters:
      wrappedStream - Stream to wrap. May be null for case where we want to write to a file backed stream only.
      Throws:
      IOException - If failed creation of backing file.

    • ensureDiskStream

      protected OutputStream ensureDiskStream() throws FileNotFoundException
      Throws:
      FileNotFoundException

    • write

      public void write(int b) throws IOException
      Specified by:
      write in class OutputStream
      Throws:
      IOException

    • write

      public void write(byte[] b, int off, int len) throws IOException
      Overrides:
      write in class OutputStream
      Throws:
      IOException

    • checkLimits

      protected void checkLimits() throws RecorderIOException
      Check any enforced limits.
      Throws:
      RecorderIOException

    • close

      public void close() throws IOException
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Overrides:
      close in class OutputStream
      Throws:
      IOException

    • closeDiskStream

      protected void closeDiskStream() throws IOException
      Throws:
      IOException

    • closeRecorder

      public void closeRecorder() throws IOException
      Throws:
      IOException

    • flush

      public void flush() throws IOException
      Specified by:
      flush in interface Flushable
      Overrides:
      flush in class OutputStream
      Throws:
      IOException

    • getReplayInputStream

      public ReplayInputStream getReplayInputStream() throws IOException
      Throws:
      IOException

    • getReplayInputStream

      public ReplayInputStream getReplayInputStream(long skip) throws IOException
      Throws:
      IOException

    • getMessageBodyReplayInputStream

      public ReplayInputStream getMessageBodyReplayInputStream() throws IOException
      Return a replay stream, cued up to begining of content
      Returns:
      An RIS.
      Throws:
      IOException

    • getSize

      public long getSize()

    • markMessageBodyBegin

      public void markMessageBodyBegin()
      Remember the current position as the start of the "message body". Useful when recording HTTP traffic as a way to start replays after the headers.

    • getMessageBodyBegin

      public long getMessageBodyBegin()
      Return stored message-body-begin-mark (which is also end-of-headers)

    • startDigest

      public void startDigest()
      Starts digesting recorded data, if a MessageDigest has been set.

    • setSha1Digest

      public void setSha1Digest()
      Convenience method for setting SHA1 digest.
      See Also:

    • setDigest

      public void setDigest(String algorithm)
      Sets a digest function which may be applied to recorded data. The difference between calling this method and setDigest(MessageDigest) is that this method tries to reuse MethodDigest instance if already allocated and of appropriate algorithm.
      Parameters:
      algorithm - Message digest algorithm to use.
      See Also:

    • setDigest

      public void setDigest(MessageDigest md)
      Sets a digest function which may be applied to recorded data. As usually only a subset of the recorded data should be fed to the digest, you must also call startDigest() to begin digesting.
      Parameters:
      md - Message digest function to use.

    • getDigestValue

      public byte[] getDigestValue()
      Return the digest value for any recorded, digested data. Call only after all data has been recorded; otherwise, the running digest state is ruined.
      Returns:
      the digest final value

    • getResponseContentLength

      public long getResponseContentLength()

    • isOpen

      public boolean isOpen()
      Returns:
      True if this ROS is open.

    • getBufferLength

      public int getBufferLength()

    • mark

      public void mark()
      When used alongside a mark-supporting RecordingInputStream, remember a position reachable by a future reset().

    • reset

      public void reset()
      When used alongside a mark-supporting RecordingInputStream, reset the position to that saved by previous mark(). Until the position again reached "new" material, none of the bytes pushed to this stream will be digested or recorded.

    • setLimits

      public void setLimits(long length, long milliseconds, long rateKBps)
      Set limits on length, time, and rate to enforce.
      Parameters:
      length -
      milliseconds -
      rateKBps -

    • resetLimits

      public void resetLimits()
      Reset limits to effectively-unlimited defaults

    • getRemainingLength

      public long getRemainingLength()
      Return number of bytes that could be recorded without hitting length limit
      Returns:
      long byte count

    • chopAtMessageBodyBegin

      public void chopAtMessageBodyBegin()
      Forget about anything past the point where the content-body starts. This is needed to support FetchHTTP's shouldFetchBody setting. See also the docs on lastTwoBytes

    • clearForReuse

      public void clearForReuse() throws IOException
      Throws:
      IOException