public class WarcHeader extends java.lang.Object
| Modifier and Type | Field and Description |
|---|---|
boolean |
bMagicIdentified
Was "WARC/" identified while looking for the version string.
|
boolean |
bMandatoryMissing
Is the header missing one of the mandatory headers.
|
boolean |
bValidVersion
Is the version recognized.
|
boolean |
bValidVersionFormat
Is the version format valid.
|
boolean |
bVersionParsed
Did the version string include between 2 and 4 substrings delimited by ".".
|
java.lang.Long |
contentLength
Content-Length converted to a
Long object, if valid. |
java.lang.String |
contentLengthStr
Content-Length field string value.
|
ContentType |
contentType
Content-Type converted to a
ContentType object, if valid. |
java.lang.String |
contentTypeStr
Content-Type field string value.
|
protected Diagnostics |
diagnostics
Diagnostics used to report diagnoses.
|
protected WarcFieldParsers |
fieldParsers
WARC field parser used.
|
byte[] |
headerBytes
Raw WARC header byte array.
|
protected java.io.ByteArrayOutputStream |
headerBytesOut
Raw WARC header output stream.
|
protected HeaderLineReader |
headerLineReader
Header line reader used to read the WARC headers.
|
protected java.util.List<HeaderLine> |
headerList
List of parsed header fields.
|
protected java.util.Map<java.lang.String,HeaderLine> |
headerMap
Map of parsed header fields.
|
protected HeaderLineReader |
lineReader
Line reader used to read version lines.
|
int |
major
Major version number from WARC header.
|
int |
minor
Minor version number from WARC header.
|
protected WarcReader |
reader
Associated WarcReader context.
|
protected int |
recordHeaderMaxSize
Max size allowed for a record header.
|
protected boolean[] |
seen
Array used for duplicate header detection.
|
protected long |
startOffset
WARC record starting offset relative to the source WARC file input
stream.
|
static boolean |
URI_LTGT
An URI with encapsulating <> characters.
|
static boolean |
URI_NAKED
An URI without encapsulating <> characters.
|
protected UriProfile |
uriProfile
URI profile.
|
int[] |
versionArr
Array based on the version string split by the "." delimiter and converted to integers.
|
java.lang.String |
versionStr
Raw version string.
|
WarcDigest |
warcBlockDigest
WARC-Block-Digest converted to a
WarcDigest object, if valid. |
java.lang.String |
warcBlockDigestStr
WARC-Block-Digest field string value.
|
java.util.List<WarcConcurrentTo> |
warcConcurrentToList
List of WARC-Concurrent-To field string values and converted
URI objects, if valid. |
WarcDate |
warcDate
WARC-Date converted to a
Date object, if valid. |
java.lang.String |
warcDateStr
WARC-Date field string value.
|
java.lang.String |
warcFilename
WARC-Filename field string value.
|
ContentType |
warcIdentifiedPayloadType
WARC-Identified-Payload-Type converted to a
ContentType object, if valid. |
java.lang.String |
warcIdentifiedPayloadTypeStr
WARC-Identified-Payload-Type field string value.
|
java.net.InetAddress |
warcInetAddress
WARC-IP-Address converted to an
InetAddress object, if valid. |
java.lang.String |
warcIpAddress
WARC-IP-Address field string value.
|
WarcDigest |
warcPayloadDigest
WARC-Payload-Digest converted to a
WarcDigest object, if valid. |
java.lang.String |
warcPayloadDigestStr
WARC-Payload-Digest field string value.
|
java.lang.Integer |
warcProfileIdx
WARC-Profile converted to an integer id, if valid.
|
java.lang.String |
warcProfileStr
WARC-Profile field string value.
|
Uri |
warcProfileUri
WARC-Profile field converted to an
Uri object, if valid. |
java.lang.String |
warcRecordIdStr
WARC-Record-Id field string value.
|
Uri |
warcRecordIdUri
WARC-Record-Id converted to an
Uri object, if valid. |
WarcDate |
warcRefersToDate
WARC-Date converted to a
Date object, if valid. |
java.lang.String |
warcRefersToDateStr
WARC-Refers-To-Date
|
java.lang.String |
warcRefersToStr
WARC-Refers-To field string value.
|
java.lang.String |
warcRefersToTargetUriStr
WARC-Refers-To-Target-URI field string value.
|
Uri |
warcRefersToTargetUriUri
WARC-Refers-To-Target-URI converted to an
Uri object, if valid. |
Uri |
warcRefersToUri
WARC-Refers-To converted to an
Uri object, if valid. |
java.lang.Integer |
warcSegmentNumber
WARC-Segment-Number converted to an
Integer object, if valid. |
java.lang.String |
warcSegmentNumberStr
WARC-Segment-Number field string value.
|
java.lang.String |
warcSegmentOriginIdStr
WARC-Segment-Origin-Id field string value.
|
Uri |
warcSegmentOriginIdUrl
WARC-Segment-Origin-Id converted to an
Uri object, if valid. |
java.lang.Long |
warcSegmentTotalLength
WARC-Segment-Total-Length converted to a
Long object, if valid. |
java.lang.String |
warcSegmentTotalLengthStr
WARC-Segment-Total-Length field string value.
|
protected UriProfile |
warcTargetUriProfile
WARC-Target-URI profile.
|
java.lang.String |
warcTargetUriStr
WARC_Target-URI field string value.
|
Uri |
warcTargetUriUri
WARC-TargetURI converted to an
Uri object, if valid. |
java.lang.Integer |
warcTruncatedIdx
WARC-Truncated converted to an integer id, if valid.
|
java.lang.String |
warcTruncatedStr
WARC-Truncated field string value.
|
java.lang.Integer |
warcTypeIdx
WARC-Type converted to an integer id, if identified.
|
java.lang.String |
warcTypeStr
WARC-Type field string value.
|
protected WarcValidatorBase |
warcValidator
Version based WARC validation.
|
java.lang.String |
warcWarcinfoIdStr
WARC-Warcinfo-Id field string value.
|
Uri |
warcWarcinfoIdUri
WARC-Warcinfo-Id converted to an
Uri object, if valid. |
| Modifier | Constructor and Description |
|---|---|
protected |
WarcHeader()
Non public constructor to allow unit testing.
|
| Modifier and Type | Method and Description |
|---|---|
void |
addHeader(HeaderLine headerLine)
Identify a (WARC) header name, validate the value and set the header.
|
HeaderLine |
addHeader(java.lang.String fieldName,
ContentType contentTypeFieldValue,
java.lang.String fieldValueStr)
Add an Content-Type header using the supplied string and object values and return
a
HeaderLine object corresponding to how the header would be read. |
HeaderLine |
addHeader(java.lang.String fieldName,
java.net.InetAddress inetAddrFieldValue,
java.lang.String fieldValueStr)
Add an InetAddress header using the supplied string and object values and return
a
HeaderLine object corresponding to how the header would be read. |
HeaderLine |
addHeader(java.lang.String fieldName,
java.lang.Integer integerFieldValue,
java.lang.String fieldValueStr)
Add an Integer header using the supplied string and object values and return
a
HeaderLine object corresponding to how the header would be read. |
HeaderLine |
addHeader(java.lang.String fieldName,
java.lang.Long longFieldValue,
java.lang.String fieldValueStr)
Add a Long header using the supplied string and object values and return
a
HeaderLine object corresponding to how the header would be read. |
HeaderLine |
addHeader(java.lang.String fieldName,
java.lang.String fieldValue)
Add a String header using the supplied string and return a
HeaderLine object corresponding to how the header would be
read. |
HeaderLine |
addHeader(java.lang.String fieldName,
java.lang.String fieldValueStr,
int dt,
java.lang.Integer integerFieldValue,
java.lang.Long longFieldValue,
WarcDigest digestFieldValue,
ContentType contentTypeFieldValue,
WarcDate dateFieldValue,
java.net.InetAddress inetAddrFieldValue,
Uri uriFieldValue)
Add a header with the supplied field name, data type and value and
return a
HeaderLine corresponding to how the header will
be read. |
HeaderLine |
addHeader(java.lang.String fieldName,
Uri uriFieldValue,
java.lang.String fieldValueStr)
Add an URI header using the supplied string and object values and return
a
HeaderLine object corresponding to how the header would be read. |
HeaderLine |
addHeader(java.lang.String fieldName,
WarcDate dateFieldValue,
java.lang.String fieldValueStr)
Add an Date header using the supplied string and object values and return
a
HeaderLine object corresponding to how the header would be read. |
HeaderLine |
addHeader(java.lang.String fieldName,
WarcDigest digestFieldValue,
java.lang.String fieldValueStr)
Add an Digest header using the supplied string and object values and return
a
HeaderLine object corresponding to how the header would be read. |
protected void |
addWarningDiagnosis(DiagnosisType type,
java.lang.String entity,
java.lang.String... information)
Add a warning diagnosis of the given type on a specific entity with
optional extra information.
|
HeaderLine |
getHeader(java.lang.String field)
Get a header line structure or null, if no header line structure is
stored with the given header name.
|
java.util.List<HeaderLine> |
getHeaderList()
Get a
List of all the headers found during parsing. |
long |
getStartOffset()
Returns the starting offset of the record in the containing WARC.
|
static WarcHeader |
initHeader(int recordHeaderMaxSize,
HeaderLineReader lineReader,
HeaderLineReader headerLineReader,
WarcFieldParsers fieldParsers,
UriProfile uriProfile,
Diagnostics diagnostics)
Create a new
WarcHeader for recreating a header object. |
static WarcHeader |
initHeader(WarcReader reader,
long startOffset,
Diagnostics diagnostics)
Create and initialize a new
WarcHeader for reading. |
static WarcHeader |
initHeader(WarcWriter writer,
Diagnostics diagnostics)
Create and initialize a new
WarcHeader for writing. |
boolean |
parseHeader(ByteCountingPushBackInputStream in)
Try to parse a WARC header and return a boolean indicating the success or
failure of this.
|
void |
parseHeaders(ByteCountingPushBackInputStream in)
Reads WARC header lines one line at a time until an empty line is
encountered.
|
boolean |
parseVersion(ByteCountingPushBackInputStream in)
Looks forward in the input stream for a valid WARC version line.
|
public static final boolean URI_LTGT
public static final boolean URI_NAKED
protected WarcReader reader
protected WarcValidatorBase warcValidator
protected Diagnostics diagnostics
protected UriProfile warcTargetUriProfile
protected UriProfile uriProfile
protected WarcFieldParsers fieldParsers
protected int recordHeaderMaxSize
protected HeaderLineReader lineReader
protected HeaderLineReader headerLineReader
protected long startOffset
public boolean bMagicIdentified
public boolean bVersionParsed
public boolean bValidVersionFormat
public boolean bValidVersion
public java.lang.String versionStr
public int[] versionArr
public int major
public int minor
protected boolean[] seen
public boolean bMandatoryMissing
public java.lang.String warcTypeStr
public java.lang.Integer warcTypeIdx
public java.lang.String warcFilename
public java.lang.String warcRecordIdStr
public Uri warcRecordIdUri
Uri object, if valid.public java.lang.String warcDateStr
public java.lang.String contentLengthStr
public java.lang.Long contentLength
Long object, if valid.public java.lang.String contentTypeStr
public ContentType contentType
ContentType object, if valid.public java.lang.String warcTruncatedStr
public java.lang.Integer warcTruncatedIdx
public java.lang.String warcIpAddress
public java.net.InetAddress warcInetAddress
InetAddress object, if valid.public java.util.List<WarcConcurrentTo> warcConcurrentToList
URI objects, if valid.public java.lang.String warcRefersToStr
public Uri warcRefersToUri
Uri object, if valid.public java.lang.String warcTargetUriStr
public Uri warcTargetUriUri
Uri object, if valid.public java.lang.String warcWarcinfoIdStr
public Uri warcWarcinfoIdUri
Uri object, if valid.public java.lang.String warcBlockDigestStr
public WarcDigest warcBlockDigest
WarcDigest object, if valid.public java.lang.String warcPayloadDigestStr
public WarcDigest warcPayloadDigest
WarcDigest object, if valid.public java.lang.String warcIdentifiedPayloadTypeStr
public ContentType warcIdentifiedPayloadType
ContentType object, if valid.public java.lang.String warcProfileStr
public Uri warcProfileUri
Uri object, if valid.
(revisit record only)public java.lang.Integer warcProfileIdx
public java.lang.String warcSegmentNumberStr
public java.lang.Integer warcSegmentNumber
Integer object, if valid.public java.lang.String warcSegmentOriginIdStr
public Uri warcSegmentOriginIdUrl
Uri object, if valid.
(continuation record only)public java.lang.String warcSegmentTotalLengthStr
public java.lang.Long warcSegmentTotalLength
Long object, if valid.
(continuation record only)public java.lang.String warcRefersToTargetUriStr
public Uri warcRefersToTargetUriUri
Uri object, if valid.public java.lang.String warcRefersToDateStr
public WarcDate warcRefersToDate
Date object, if valid.protected java.io.ByteArrayOutputStream headerBytesOut
public byte[] headerBytes
protected java.util.List<HeaderLine> headerList
protected java.util.Map<java.lang.String,HeaderLine> headerMap
protected WarcHeader()
public static WarcHeader initHeader(int recordHeaderMaxSize, HeaderLineReader lineReader, HeaderLineReader headerLineReader, WarcFieldParsers fieldParsers, UriProfile uriProfile, Diagnostics diagnostics)
WarcHeader for recreating a header object.recordHeaderMaxSize - max size allowed for a record headerlineReader - line reader used to read version linesheaderLineReader - header line reader used to read the WARC headersfieldParsers - parsers used for the individual typesuriProfile - uri profile used to validate urlsdiagnostics - diagnostics object used by parserWarcHeader prepared for reanimationpublic static WarcHeader initHeader(WarcWriter writer, Diagnostics diagnostics)
WarcHeader for writing.writer - writer which shall be useddiagnostics - diagnostics object used by writerWarcHeader prepared for writingpublic static WarcHeader initHeader(WarcReader reader, long startOffset, Diagnostics diagnostics)
WarcHeader for reading.reader - reader which shall be usedstartOffset - start offset of headerdiagnostics - diagnostics object used by readerWarcHeader prepared for readingprotected void addWarningDiagnosis(DiagnosisType type, java.lang.String entity, java.lang.String... information)
type - diagnosis typeentity - entity examinedinformation - optional extra informationpublic long getStartOffset()
public boolean parseHeader(ByteCountingPushBackInputStream in) throws java.io.IOException
in - input stream with WARC datajava.io.IOException - if an I/O exception occurs while parsing for a headerpublic boolean parseVersion(ByteCountingPushBackInputStream in) throws java.io.IOException
in - data input streamjava.io.IOException - if an error occurs while reading version datapublic void parseHeaders(ByteCountingPushBackInputStream in) throws java.io.IOException
in - header input streamjava.io.IOException - if an error occurs while reading the WARC headerpublic void addHeader(HeaderLine headerLine)
headerLine - the headerLinepublic java.util.List<HeaderLine> getHeaderList()
List of all the headers found during parsing.List of HeaderLinepublic HeaderLine getHeader(java.lang.String field)
field - header nameHeaderLine structure or nullpublic HeaderLine addHeader(java.lang.String fieldName, java.lang.String fieldValue)
HeaderLine object corresponding to how the header would be
read.fieldName - name of field to addfieldValue - field value stringHeaderLine object corresponding to what would have been readpublic HeaderLine addHeader(java.lang.String fieldName, java.lang.Integer integerFieldValue, java.lang.String fieldValueStr)
HeaderLine object corresponding to how the header would be read.
If both string and object values are not null they are used as is.
If the string value is null and the object is not null,
the object's toString method is called.
If the object is null and the string is not null, the string is parsed
and validated resulting in an object, if valid.fieldName - name of field to addintegerFieldValue - Integer field value objectfieldValueStr - Integer field value stringHeaderLine object corresponding to what would have been readpublic HeaderLine addHeader(java.lang.String fieldName, java.lang.Long longFieldValue, java.lang.String fieldValueStr)
HeaderLine object corresponding to how the header would be read.
If both string and object values are not null they are used as is.
If the string value is null and the object is not null,
the object's toString method is called.
If the object is null and the string is not null, the string is parsed
and validated resulting in an object, if valid.fieldName - name of field to addlongFieldValue - Long field value objectfieldValueStr - Long field value stringHeaderLine object corresponding to what would have been readpublic HeaderLine addHeader(java.lang.String fieldName, WarcDigest digestFieldValue, java.lang.String fieldValueStr)
HeaderLine object corresponding to how the header would be read.
If both string and object values are not null they are used as is.
If the string value is null and the object is not null,
the object's toString method is called.
If the object is null and the string is not null, the string is parsed
and validated resulting in an object, if valid.fieldName - name of field to adddigestFieldValue - Digest field value objectfieldValueStr - Digest field value stringHeaderLine object corresponding to what would have been readpublic HeaderLine addHeader(java.lang.String fieldName, ContentType contentTypeFieldValue, java.lang.String fieldValueStr)
HeaderLine object corresponding to how the header would be read.
If both string and object values are not null they are used as is.
If the string value is null and the object is not null,
the object's toString method is called.
If the object is null and the string is not null, the string is parsed
and validated resulting in an object, if valid.fieldName - name of field to addcontentTypeFieldValue - ContentType field value objectfieldValueStr - Content-Type field value stringHeaderLine object corresponding to what would have been readpublic HeaderLine addHeader(java.lang.String fieldName, WarcDate dateFieldValue, java.lang.String fieldValueStr)
HeaderLine object corresponding to how the header would be read.
If both string and object values are not null they are used as is.
If the string value is null and the object is not null,
the object's toString method is called.
If the object is null and the string is not null, the string is parsed
and validated resulting in an object, if valid.fieldName - name of field to adddateFieldValue - Date field value objectfieldValueStr - Date field value stringHeaderLine object corresponding to what would have been readpublic HeaderLine addHeader(java.lang.String fieldName, java.net.InetAddress inetAddrFieldValue, java.lang.String fieldValueStr)
HeaderLine object corresponding to how the header would be read.
If both string and object values are not null they are used as is.
If the string value is null and the object is not null,
the object's toString method is called.
If the object is null and the string is not null, the string is parsed
and validated resulting in an object, if valid.fieldName - name of field to addinetAddrFieldValue - InetAddress field value objectfieldValueStr - IP-Address field value stringHeaderLine object corresponding to what would have been readpublic HeaderLine addHeader(java.lang.String fieldName, Uri uriFieldValue, java.lang.String fieldValueStr)
HeaderLine object corresponding to how the header would be read.
If both string and object values are not null they are used as is.
If the string value is null and the object is not null,
the object's toString method is called.
If the object is null and the string is not null, the string is parsed
and validated resulting in an object, if valid.fieldName - name of field to adduriFieldValue - URI field value objectfieldValueStr - URI field value stringHeaderLine object corresponding to what would have been readpublic HeaderLine addHeader(java.lang.String fieldName, java.lang.String fieldValueStr, int dt, java.lang.Integer integerFieldValue, java.lang.Long longFieldValue, WarcDigest digestFieldValue, ContentType contentTypeFieldValue, WarcDate dateFieldValue, java.net.InetAddress inetAddrFieldValue, Uri uriFieldValue)
HeaderLine corresponding to how the header will
be read. The data type is validated against the field data type.
The values used are the field value string and the parameter
corresponding to the data type.fieldName - header field namefieldValueStr - field value in string formdt - data type of the field value string when converted to an objectintegerFieldValue - Integer object field valuelongFieldValue - Long object field valuedigestFieldValue - Digest object field valuecontentTypeFieldValue - ContentType object field valuedateFieldValue - Date object field valueinetAddrFieldValue - InetAddress object field valueuriFieldValue - URI object field valueHeaderLine object corresponding to what would have been readCopyright © 2011–2023. All rights reserved.