org.archive.format.text.charset

Class CharsetDetector

java.lang.Object

org.archive.format.text.charset.CharsetDetector

Direct Known Subclasses:

RotatingCharsetDetector, StandardCharsetDetector

public abstract class CharsetDetector
extends java.lang.Object

Abstract class containing common methods for determining the character encoding of a text Resource, most of which should be refactored into a Util package.

Author:

brad

Field Summary

Fields

Modifier and Type	Field and Description
protected static java.lang.String	CHARSET_TOKEN
static java.lang.String	DEFAULT_CHARSET the default charset name to use when giving up
protected static java.lang.String	HTTP_CONTENT_TYPE_HEADER
protected static int	MAX_CHARSET_READAHEAD

Constructor Summary

Constructors

Constructor and Description
CharsetDetector()

Method Summary

Modifier and Type	Method and Description
protected java.lang.String	contentTypeToCharset(java.lang.String contentType)
static java.lang.String	findMetaContentType(java.lang.String pageSample)
abstract java.lang.String	getCharset(byte[] buffer, int len, HttpHeaders headers)
protected java.lang.String	getCharsetFromBytes(byte[] buffer, int len) Attempts to figure out the character set of the document using the excellent juniversalchardet library.
protected java.lang.String	getCharsetFromHeaders(HttpHeaders headers) Attempt to divine the character encoding of the document from the Content-Type HTTP header (with a "charset=")
protected java.lang.String	getCharsetFromMeta(byte[] buffer, int len) Attempt to find a META tag in the HTML that hints at the character set used to write the document.
protected boolean	isCharsetSupported(java.lang.String charsetName)
protected java.lang.String	mapCharset(java.lang.String orig)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAX_CHARSET_READAHEAD

protected static final int MAX_CHARSET_READAHEAD

See Also:

Constant Field Values

CHARSET_TOKEN

protected static final java.lang.String CHARSET_TOKEN

See Also:

Constant Field Values

HTTP_CONTENT_TYPE_HEADER

protected static final java.lang.String HTTP_CONTENT_TYPE_HEADER

See Also:

Constant Field Values

DEFAULT_CHARSET

public static final java.lang.String DEFAULT_CHARSET

the default charset name to use when giving up

See Also:

Constant Field Values

Constructor Detail

CharsetDetector

public CharsetDetector()

Method Detail

isCharsetSupported

protected boolean isCharsetSupported(java.lang.String charsetName)

mapCharset

protected java.lang.String mapCharset(java.lang.String orig)

contentTypeToCharset

protected java.lang.String contentTypeToCharset(java.lang.String contentType)

getCharsetFromHeaders

protected java.lang.String getCharsetFromHeaders(HttpHeaders headers)
                                          throws java.io.IOException

Attempt to divine the character encoding of the document from the Content-Type HTTP header (with a "charset=")

Parameters:

resource -

Returns:

String character set found or null if the header was not present

Throws:

java.io.IOException

getCharsetFromMeta

protected java.lang.String getCharsetFromMeta(byte[] buffer,
                                              int len)
                                       throws java.io.IOException

Attempt to find a META tag in the HTML that hints at the character set used to write the document.

Parameters:

resource -

Returns:

String character set found from META tags in the HTML

Throws:

java.io.IOException

findMetaContentType

public static java.lang.String findMetaContentType(java.lang.String pageSample)

getCharsetFromBytes

protected java.lang.String getCharsetFromBytes(byte[] buffer,
                                               int len)
                                        throws java.io.IOException

Attempts to figure out the character set of the document using the excellent juniversalchardet library.

Parameters:

resource -

Returns:

String character encoding found, or null if nothing looked good.

Throws:

java.io.IOException

getCharset

public abstract java.lang.String getCharset(byte[] buffer,
                                            int len,
                                            HttpHeaders headers)
                                     throws java.io.IOException

Parameters:

resource - (presumably text) Resource to determine the charset

request - WaybackRequest which may contain additional hints to processing

Returns:

String charset name for the Resource

Throws:

java.io.IOException - if there are problems reading the Resource