org.archive.io

Class RecordingOutputStream

java.lang.Object

java.io.OutputStream

org.archive.io.RecordingOutputStream

All Implemented Interfaces:

Closeable, Flushable, AutoCloseable

public class RecordingOutputStream
extends OutputStream

An output stream that records all writes to wrapped output stream. A RecordingOutputStream can be wrapped around any other OutputStream to record all bytes written to it. You can then request a ReplayInputStream to read those bytes.

The RecordingOutputStream uses an in-memory buffer and backing disk file to allow it to record streams of arbitrary length limited only by available disk space.

As long as the stream recorded is smaller than the in-memory buffer, no disk access will occur.

Recorded content can be recovered as a ReplayInputStream (via getReplayInputStream() or, for only the content after the content-begin-mark is set, getContentReplayInputStream() ) or as a ReplayCharSequence (via getReplayCharSequence()).

This class is also used as a straight output stream by RecordingInputStream to which it records all reads. RecordingInputStream is exploiting the file backed buffer facility of this class passing null for the stream to wrap. TODO: Make a FileBackedOutputStream class that is subclassed by RecordingInputStream.

Author:

gojomo

Field Summary

Fields

Modifier and Type	Field and Description
protected String	backingFilename
protected OutputStream	diskStream
protected int[]	lastTwoBytes While messageBodyBeginMark is not set, the last two bytes seen.
protected static Logger	logger
protected static long	MAX_HEADER_MATERIAL Maximum amount of header material to accept without the content body beginning -- if more, throw a RecorderTooMuchHeaderException.
protected long	maxLength maximum length of material to record before throwing exception
protected long	maxRateBytesPerMs maximum rate to record (adds delays to hit target rate)
protected long	messageBodyBeginMark When recording HTTP, where the content-body starts.
protected long	size Size of recording.
protected long	startTime time recording begins for timeout, rate calculations
protected long	timeoutMs maximum time to record before throwing exception

Constructor Summary

Constructors

Constructor and Description
RecordingOutputStream(int bufferSize, String backingFilename) Create a new RecordingOutputStream.

Method Summary

Methods

Modifier and Type	Method and Description
protected void	checkLimits() Check any enforced limits.
void	chopAtMessageBodyBegin() Forget about anything past the point where the content-body starts.
void	clearForReuse()
void	close()
protected void	closeDiskStream()
void	closeRecorder()
protected OutputStream	ensureDiskStream()
void	flush()
int	getBufferLength()
byte[]	getDigestValue() Return the digest value for any recorded, digested data.
long	getMessageBodyBegin() Return stored message-body-begin-mark (which is also end-of-headers)
ReplayInputStream	getMessageBodyReplayInputStream() Return a replay stream, cued up to begining of content
long	getRemainingLength() Return number of bytes that could be recorded without hitting length limit
ReplayInputStream	getReplayInputStream()
ReplayInputStream	getReplayInputStream(long skip)
long	getResponseContentLength()
long	getSize()
boolean	isOpen()
void	mark() When used alongside a mark-supporting RecordingInputStream, remember a position reachable by a future reset().
void	markMessageBodyBegin() Remember the current position as the start of the "message body".
void	open() Wrap the given stream, both recording and passing along any data written to this RecordingOutputStream.
void	open(OutputStream wrappedStream) Wrap the given stream, both recording and passing along any data written to this RecordingOutputStream.
void	reset() When used alongside a mark-supporting RecordingInputStream, reset the position to that saved by previous mark().
void	resetLimits() Reset limits to effectively-unlimited defaults
void	setDigest(MessageDigest md) Sets a digest function which may be applied to recorded data.
void	setDigest(String algorithm) Sets a digest function which may be applied to recorded data.
void	setLimits(long length, long milliseconds, long rateKBps) Set limits on length, time, and rate to enforce.
void	setSha1Digest() Convenience method for setting SHA1 digest.
void	startDigest() Starts digesting recorded data, if a MessageDigest has been set.
void	write(byte[] b, int off, int len)
void	write(int b)

Methods inherited from class java.io.OutputStream

write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

logger

protected static Logger logger

size

protected long size

Size of recording. Later passed to ReplayInputStream on creation. It uses it to know when EOS.

backingFilename

protected String backingFilename

diskStream

protected OutputStream diskStream

MAX_HEADER_MATERIAL

protected static final long MAX_HEADER_MATERIAL

Maximum amount of header material to accept without the content body beginning -- if more, throw a RecorderTooMuchHeaderException. TODO: make configurable? make smaller?

See Also:

Constant Field Values

maxLength

protected long maxLength

maximum length of material to record before throwing exception

timeoutMs

protected long timeoutMs

maximum time to record before throwing exception

maxRateBytesPerMs

protected long maxRateBytesPerMs

maximum rate to record (adds delays to hit target rate)

startTime

protected long startTime

time recording begins for timeout, rate calculations

messageBodyBeginMark

protected long messageBodyBeginMark

When recording HTTP, where the content-body starts.

lastTwoBytes

protected int[] lastTwoBytes

While messageBodyBeginMark is not set, the last two bytes seen.

This class does automatic detection of http message body begin (i.e. end of http headers). Unfortunately httpcomponents did not want to add functionality to help us with this, see https://issues.apache.org/jira/browse/HTTPCORE-325

It works like this: while messageBodyBeginMark is not set, we remember the last two bytes seen, and look at each byte we write. If the lastTwoBytes+currentByte is "\n\r\n", or lastTwoBytes[1]+currentByte is "\n\n" then we call markMessageBodyBegin() at the position after currentByte.

An assumption here is that protocols other than http don't have headers, and for those protocols the user of this class will call markMessageBodyBegin() at position 0 before writing anything.

Constructor Detail

RecordingOutputStream

public RecordingOutputStream(int bufferSize,
                     String backingFilename)

Create a new RecordingOutputStream.

Parameters:

bufferSize - Buffer size to use.

backingFilename - Name of backing file to use.

Method Detail

open

public void open()
          throws IOException

Wrap the given stream, both recording and passing along any data written to this RecordingOutputStream.

Throws:

IOException - If failed creation of backing file.

open

public void open(OutputStream wrappedStream)
          throws IOException

Wrap the given stream, both recording and passing along any data written to this RecordingOutputStream.

Parameters:

wrappedStream - Stream to wrap. May be null for case where we want to write to a file backed stream only.

Throws:

IOException - If failed creation of backing file.

ensureDiskStream

protected OutputStream ensureDiskStream()
                                 throws FileNotFoundException

Throws:

FileNotFoundException

write

public void write(int b)
           throws IOException

Specified by:

write in class OutputStream

Throws:

IOException

write

public void write(byte[] b,
         int off,
         int len)
           throws IOException

Overrides:

write in class OutputStream

Throws:

IOException

checkLimits

protected void checkLimits()
                    throws RecorderIOException

Check any enforced limits.

Throws:

RecorderIOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class OutputStream

Throws:

IOException

closeDiskStream

protected void closeDiskStream()
                        throws IOException

Throws:

IOException

closeRecorder

public void closeRecorder()
                   throws IOException

Throws:

IOException

flush

public void flush()
           throws IOException

Specified by:

flush in interface Flushable

Overrides:

flush in class OutputStream

Throws:

IOException

getReplayInputStream

public ReplayInputStream getReplayInputStream()
                                       throws IOException

Throws:

IOException

getReplayInputStream

public ReplayInputStream getReplayInputStream(long skip)
                                       throws IOException

Throws:

IOException

getMessageBodyReplayInputStream

public ReplayInputStream getMessageBodyReplayInputStream()
                                                  throws IOException

Return a replay stream, cued up to begining of content

Returns:

An RIS.

Throws:

IOException

getSize

public long getSize()

markMessageBodyBegin

public void markMessageBodyBegin()

Remember the current position as the start of the "message body". Useful when recording HTTP traffic as a way to start replays after the headers.

getMessageBodyBegin

public long getMessageBodyBegin()

Return stored message-body-begin-mark (which is also end-of-headers)

startDigest

public void startDigest()

Starts digesting recorded data, if a MessageDigest has been set.

setSha1Digest

public void setSha1Digest()

Convenience method for setting SHA1 digest.

See Also:

setDigest(String)

setDigest

public void setDigest(String algorithm)

Sets a digest function which may be applied to recorded data. The difference between calling this method and setDigest(MessageDigest) is that this method tries to reuse MethodDigest instance if already allocated and of appropriate algorithm.

Parameters:

algorithm - Message digest algorithm to use.

See Also:

setDigest(MessageDigest)

setDigest

public void setDigest(MessageDigest md)

Sets a digest function which may be applied to recorded data. As usually only a subset of the recorded data should be fed to the digest, you must also call startDigest() to begin digesting.

Parameters:

md - Message digest function to use.

getDigestValue

public byte[] getDigestValue()

Return the digest value for any recorded, digested data. Call only after all data has been recorded; otherwise, the running digest state is ruined.

Returns:

the digest final value

getResponseContentLength

public long getResponseContentLength()

isOpen

public boolean isOpen()

Returns:

True if this ROS is open.

getBufferLength

public int getBufferLength()

mark

public void mark()

When used alongside a mark-supporting RecordingInputStream, remember a position reachable by a future reset().

reset

public void reset()

When used alongside a mark-supporting RecordingInputStream, reset the position to that saved by previous mark(). Until the position again reached "new" material, none of the bytes pushed to this stream will be digested or recorded.

setLimits

public void setLimits(long length,
             long milliseconds,
             long rateKBps)

Set limits on length, time, and rate to enforce.

Parameters:

length -

milliseconds -

rateKBps -

resetLimits

public void resetLimits()

Reset limits to effectively-unlimited defaults

getRemainingLength

public long getRemainingLength()

Return number of bytes that could be recorded without hitting length limit

Returns:

long byte count

chopAtMessageBodyBegin

public void chopAtMessageBodyBegin()

Forget about anything past the point where the content-body starts. This is needed to support FetchHTTP's shouldFetchBody setting. See also the docs on lastTwoBytes

clearForReuse

public void clearForReuse()
                   throws IOException

Throws:

IOException