org.archive.util

Class Recorder

java.lang.Object

org.archive.util.Recorder

public class Recorder
extends Object

Pairs together a RecordingInputStream and RecordingOutputStream to capture exactly a single HTTP transaction. Initially only supports HTTP/1.0 (one request, one response per stream) Call markContentBegin() to demarc the transition between HTTP header and body.

Author:

gojomo

Field Summary

Fields

Modifier and Type	Field and Description
protected String	characterEncoding recording-input (ris) content character encoding.
protected Charset	charset Charset to use for CharSequence provision.
protected String	contentEncoding recording-input (ris) entity content-encoding (eg gzip, deflate), if any
protected static ThreadLocal<Recorder>	currentRecorder
protected boolean	inputIsChunked whether recording-input (ris) message-body is chunked
protected static Logger	logger
protected static Set<String>	SUPPORTED_ENCODINGS

Constructor Summary

Constructors

Constructor and Description
Recorder(File file, int outBufferSize, int inBufferSize)
Recorder(File tempDir, String backingFilenameBase) Create an HttpRecorder.
Recorder(File tempDir, String backingFilenameBase, int outBufferSize, int inBufferSize) Create an HttpRecorder.

Method Summary

Methods

Modifier and Type	Method and Description
protected static int	calcRecommendedCharBufferSize(RecordingInputStream inStream) Calculate a recommended size for an in-memory decoded-character buffer of this content.
void	cleanup() Cleanup backing files.
void	close() Close all streams.
void	closeRecorders() Close both input and output recorders.
void	copyContentBodyTo(File tempFile)
void	endReplays()
Charset	getCharset()
String	getContentEncoding()
ReplayCharSequence	getContentReplayCharSequence()
ReplayCharSequence	getContentReplayCharSequence(Charset requestedCharset)
InputStream	getContentReplayInputStream() Get a replay cued up for the 'content' (after all leading headers)
String	getContentReplayPrefixString(int size) Return a short prefix of the presumed-textual content as a String.
String	getContentReplayPrefixString(int size, Charset cs) Return a short prefix of the presumed-textual content as a String.
InputStream	getEntityReplayInputStream() Get a raw replay of the 'entity'.
static Recorder	getHttpRecorder() Get the current threads' HttpRecorder.
ReplayInputStream	getMessageBodyReplayInputStream() Get a raw replay of the 'message-body'.
RecordingInputStream	getRecordedInput() Return the internal RecordingInputStream
RecordingOutputStream	getRecordedOutput()
ReplayCharSequence	getReplayCharSequence() Deprecated. use getContentReplayCharSequence
ReplayInputStream	getReplayInputStream() Get a raw replay of all recorded data (including, for example, HTTP protocol headers)
long	getResponseContentLength()
InputStream	inputWrap(InputStream is) Wrap the provided stream with the internal RecordingInputStream open() throws an exception if RecordingInputStream is already open.
void	markContentBegin() Mark current position as the point where the HTTP headers end.
OutputStream	outputWrap(OutputStream os) Wrap the provided stream with the internal RecordingOutputStream open() throws an exception if RecordingOutputStream is already open.
void	setCharset(Charset cs)
void	setContentEncoding(String contentEncoding)
static void	setHttpRecorder(Recorder httpRecorder)
void	setInputIsChunked(boolean chunked)
static Recorder	wrapInputStreamWithHttpRecord(File dir, String basename, InputStream in, String encoding) Record the input stream for later playback by an extractor, etc.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

logger

protected static Logger logger

characterEncoding

protected String characterEncoding

recording-input (ris) content character encoding.

charset

protected Charset charset

Charset to use for CharSequence provision. Will be UTF-8 if no encoding ever requested; a Charset matching above characterEncoding if possible; ISO_8859 if above characterEncoding is unsatisfiable. TODO: unify to UTF-8 for unspecified and bad-specified cases? (current behavior is for consistency with our prior but perhaps not optimal behavior)

inputIsChunked

protected boolean inputIsChunked

whether recording-input (ris) message-body is chunked

contentEncoding

protected String contentEncoding

recording-input (ris) entity content-encoding (eg gzip, deflate), if any

currentRecorder

protected static ThreadLocal<Recorder> currentRecorder

SUPPORTED_ENCODINGS

protected static Set<String> SUPPORTED_ENCODINGS

Constructor Detail

Recorder

public Recorder(File tempDir,
        String backingFilenameBase,
        int outBufferSize,
        int inBufferSize)

Create an HttpRecorder.

Parameters:

tempDir - Directory into which we drop backing files for recorded input and output.

backingFilenameBase - Backing filename base to which we'll append suffices ris for recorded input stream and ros for recorded output stream.

outBufferSize - Size of output buffer to use.

inBufferSize - Size of input buffer to use.

Recorder

public Recorder(File file,
        int outBufferSize,
        int inBufferSize)

Recorder

public Recorder(File tempDir,
        String backingFilenameBase)

Create an HttpRecorder.

Parameters:

tempDir - Directory into which we drop backing files for recorded input and output.

backingFilenameBase - Backing filename base to which we'll append suffices ris for recorded input stream and ros for recorded output stream.

Method Detail

inputWrap

public InputStream inputWrap(InputStream is)
                      throws IOException

Wrap the provided stream with the internal RecordingInputStream open() throws an exception if RecordingInputStream is already open.

Parameters:

is - InputStream to wrap.

Returns:

The input stream wrapper which itself is an input stream. Pass this in place of the passed stream so input can be recorded.

Throws:

IOException

outputWrap

public OutputStream outputWrap(OutputStream os)
                        throws IOException

Wrap the provided stream with the internal RecordingOutputStream open() throws an exception if RecordingOutputStream is already open.

Parameters:

os - The output stream to wrap.

Returns:

The output stream wrapper which is itself an output stream. Pass this in place of the passed stream so output can be recorded.

Throws:

IOException

close

public void close()

Close all streams.

getRecordedInput

public RecordingInputStream getRecordedInput()

Return the internal RecordingInputStream

Returns:

A RIS.

getRecordedOutput

public RecordingOutputStream getRecordedOutput()

Returns:

The RecordingOutputStream.

markContentBegin

public void markContentBegin()

Mark current position as the point where the HTTP headers end.

getResponseContentLength

public long getResponseContentLength()

closeRecorders

public void closeRecorders()

Close both input and output recorders. Recorders are the output streams to which we are recording. close() closes the stream that is being recorded and the recorder. This method explicitly closes the recorder only.

cleanup

public void cleanup()

Cleanup backing files. Call when completely done w/ recorder. Removes any backing files that may have been dropped.

setHttpRecorder

public static void setHttpRecorder(Recorder httpRecorder)

getHttpRecorder

public static Recorder getHttpRecorder()

Get the current threads' HttpRecorder.

Returns:

This threads' HttpRecorder. Returns null if can't find a HttpRecorder in current instance.

setCharset

public void setCharset(Charset cs)

Parameters:

characterEncoding - Character encoding of input recording.

getCharset

public Charset getCharset()

Returns:

effective Charset of input recording

setInputIsChunked

public void setInputIsChunked(boolean chunked)

Parameters:

characterEncoding - Character encoding of input recording.

setContentEncoding

public void setContentEncoding(String contentEncoding)

Parameters:

contentEncoding - declared content-encoding of input recording.

getContentEncoding

public String getContentEncoding()

Returns:

Returns the characterEncoding.

getReplayCharSequence

public ReplayCharSequence getReplayCharSequence()
                                         throws IOException

Deprecated. use getContentReplayCharSequence

Returns:

Throws:

IOException

getContentReplayCharSequence

public ReplayCharSequence getContentReplayCharSequence()
                                                throws IOException

Returns:

A ReplayCharSequence. Caller may call ReplayCharSequence.close() when finished. However, in heritrix, the ReplayCharSequence is closed automatically when url processing has finished; in that context it's preferable not to close, so that processors can reuse the same instance.

Throws:

IOException

See Also:

#endReplays()}

getContentReplayCharSequence

public ReplayCharSequence getContentReplayCharSequence(Charset requestedCharset)
                                                throws IOException

Parameters:

characterEncoding - Encoding of recorded stream.

Returns:

A ReplayCharSequence Will return null if an IOException. Call close on returned RCS when done.

Throws:

IOException

calcRecommendedCharBufferSize

protected static int calcRecommendedCharBufferSize(RecordingInputStream inStream)

Calculate a recommended size for an in-memory decoded-character buffer of this content. We seek a size that is itself no larger (in 2-byte chars) than the memory already used by the RecordingInputStream's internal raw byte buffer, and also no larger than likely necessary. So, we take the minimum of the actual recorded byte size and the RecordingInputStream's max buffer size.

Parameters:

inStream -

Returns:

int length for in-memory decoded-character buffer

getReplayInputStream

public ReplayInputStream getReplayInputStream()
                                       throws IOException

Get a raw replay of all recorded data (including, for example, HTTP protocol headers)

Returns:

A replay input stream.

Throws:

IOException

getMessageBodyReplayInputStream

public ReplayInputStream getMessageBodyReplayInputStream()
                                                  throws IOException

Get a raw replay of the 'message-body'. For the common case of HTTP, this is the raw, possibly chunked-transfer-encoded message contents not including the leading headers.

Returns:

A replay input stream.

Throws:

IOException

getEntityReplayInputStream

public InputStream getEntityReplayInputStream()
                                       throws IOException

Get a raw replay of the 'entity'. For the common case of HTTP, this is the message-body after any (usually-unnecessary) transfer-decoding but before any content-encoding (eg gzip) decoding

Returns:

A replay input stream.

Throws:

IOException

getContentReplayInputStream

public InputStream getContentReplayInputStream()
                                        throws IOException

Get a replay cued up for the 'content' (after all leading headers)

Returns:

A replay input stream.

Throws:

IOException

getContentReplayPrefixString

public String getContentReplayPrefixString(int size)

Return a short prefix of the presumed-textual content as a String.

Parameters:

size - max length of String to return

Returns:

String prefix, or empty String (with logged exception) on any error

getContentReplayPrefixString

public String getContentReplayPrefixString(int size,
                                  Charset cs)

Return a short prefix of the presumed-textual content as a String.

Parameters:

size - max length of String to return

Returns:

String prefix, or empty String (with logged exception) on any error

copyContentBodyTo

public void copyContentBodyTo(File tempFile)
                       throws IOException

Parameters:

tempFile -

Throws:

IOException

wrapInputStreamWithHttpRecord

public static Recorder wrapInputStreamWithHttpRecord(File dir,
                                     String basename,
                                     InputStream in,
                                     String encoding)
                                              throws IOException

Record the input stream for later playback by an extractor, etc. This is convenience method used to setup an artificial HttpRecorder scenario used in unit tests, etc.

Parameters:

dir - Directory to write backing file to.

basename - of what we're recording.

in - Stream to read.

encoding - Stream encoding.

Returns:

An Recorder.

Throws:

IOException

endReplays

public void endReplays()