public abstract class WarcWriter extends java.lang.Object implements java.io.Closeable
| Modifier and Type | Field and Description |
|---|---|
protected boolean |
bExceptionOnContentLengthMismatch
Configuration for throwing exception on content-length mismatch.
|
Diagnostics<Diagnosis> |
diagnostics
Writer level errors and warnings or when writing byte headers.
|
protected WarcFieldParsers |
fieldParsers
WARC field parser used.
|
protected WarcHeader |
header
Current WARC header written.
|
protected java.lang.Long |
headerContentLength
Content-Length from the WARC header.
|
protected java.io.OutputStream |
out
Outputstream used to write WARC records.
|
protected long |
payloadWrittenTotal
Total bytes written for current record payload.
|
protected static int |
S_HEADER_WRITTEN
State after header has been written.
|
protected static int |
S_INIT
State after writer has been constructed and before records have been written.
|
protected static int |
S_PAYLOAD_WRITTEN
State after payload has been written.
|
protected static int |
S_RECORD_CLOSED
State after record has been closed.
|
protected int |
state
Current state of writer.
|
protected byte[] |
stream_copy_buffer
Buffer used by streamPayload() to copy from one stream to another.
|
protected UriProfile |
uriProfile
URI profile.
|
protected UriProfile |
warcTargetUriProfile
WARC-Target-URI profile.
|
| Constructor and Description |
|---|
WarcWriter() |
| Modifier and Type | Method and Description |
|---|---|
abstract void |
close()
Close WARC writer and free its resources.
|
protected void |
closeRecord_impl()
Closes the WARC record by writing two newlines and comparing the amount of
payload data streamed with the content-length supplied with the header.
|
abstract void |
closeRecord()
Close the WARC record in an implementation specific way.
|
boolean |
exceptionOnContentLengthMismatch()
Does this writer throw an exception if the content-length does not match
the payload amount written.
|
UriProfile |
getUriProfile()
Get the URI profile used to validate URIs.
|
UriProfile |
getWarcTargetUriProfile()
Get the URI profile used to validate WARC-Target URIs.
|
protected void |
init()
Method used to initialize a readers internal state.
|
abstract boolean |
isCompressed()
Is this writer producing compressed output.
|
void |
setExceptionOnContentLengthMismatch(boolean enabled)
Tell the writer what to do in case of mismatch between content-length
and amount payload written.
|
void |
setUriProfile(UriProfile uriProfile)
Set the URI profile used to validate URIs.
|
void |
setWarcTargetUriProfile(UriProfile uriProfile)
Set the URI profile used to validate WARC-Target URIs.
|
long |
streamPayload(java.io.InputStream in)
Stream the content of an input stream to the payload content.
|
protected byte[] |
writeHeader_impl(WarcRecord record)
Write a WARC header to the WARC output stream.
|
abstract byte[] |
writeHeader(WarcRecord record)
Write a WARC header to the WARC output stream.
|
long |
writePayload(byte[] b)
Append the content of a byte array to the payload content.
|
long |
writePayload(byte[] b,
int offset,
int len)
Append the partial content of a byte array to the payload content.
|
void |
writeRawHeader(byte[] header_bytes,
java.lang.Long contentLength)
Write a raw WARC header to the WARC output stream.
|
protected static final int S_INIT
protected static final int S_HEADER_WRITTEN
protected static final int S_PAYLOAD_WRITTEN
protected static final int S_RECORD_CLOSED
protected UriProfile warcTargetUriProfile
protected UriProfile uriProfile
protected WarcFieldParsers fieldParsers
protected byte[] stream_copy_buffer
protected boolean bExceptionOnContentLengthMismatch
public final Diagnostics<Diagnosis> diagnostics
protected int state
protected java.io.OutputStream out
protected WarcHeader header
protected java.lang.Long headerContentLength
protected long payloadWrittenTotal
public WarcWriter()
protected void init()
public abstract boolean isCompressed()
public void setWarcTargetUriProfile(UriProfile uriProfile)
uriProfile - URI profile to usepublic UriProfile getWarcTargetUriProfile()
public void setUriProfile(UriProfile uriProfile)
uriProfile - URI profile to usepublic UriProfile getUriProfile()
public boolean exceptionOnContentLengthMismatch()
public void setExceptionOnContentLengthMismatch(boolean enabled)
enabled - boolean indicating exception throwing on/offpublic abstract void close() throws java.io.IOException
close in interface java.io.Closeableclose in interface java.lang.AutoCloseablejava.io.IOException - if an I/O exception occurs while closing the writerpublic abstract void closeRecord() throws java.io.IOException
java.io.IOException - if an I/O exception occurs while closing the recordprotected void closeRecord_impl() throws java.io.IOException
java.io.IOException - if an I/O exception occurs while closing the recordpublic void writeRawHeader(byte[] header_bytes, java.lang.Long contentLength) throws java.io.IOException
header_bytes - raw WARC header to outputcontentLength - the expected content-length to be written and validatedjava.io.IOException - if an I/O exception occurs while writing header datapublic abstract byte[] writeHeader(WarcRecord record) throws java.io.IOException
record - WARC record to outputjava.io.IOException - if an I/O exception occurs while writing header dataprotected byte[] writeHeader_impl(WarcRecord record) throws java.io.IOException
record - WARC record to outputjava.io.IOException - if an I/O exception occurs while writing header datapublic long streamPayload(java.io.InputStream in) throws java.io.IOException
in - input stream containing payload datajava.io.IOException - if an I/O exception occurs while writing payload datapublic long writePayload(byte[] b) throws java.io.IOException
b - byte array with data to be writtenjava.io.IOException - if an I/O exception occurs while writing payload datapublic long writePayload(byte[] b, int offset, int len) throws java.io.IOException
b - byte array with partial data to be writtenoffset - offset to data to be writtenlen - length of data to be writtenjava.io.IOException - if an I/O exception occurs while writing payload dataCopyright © 2011–2022. All rights reserved.