Package org.apache.tika.ml.junkdetect

Class JunkFilterEncodingDetector

java.lang.Object

org.apache.tika.ml.junkdetect.JunkFilterEncodingDetector

All Implemented Interfaces:

Serializable, org.apache.tika.config.SelfConfiguring, org.apache.tika.detect.EncodingDetector, org.apache.tika.detect.MetaEncodingDetector

public class JunkFilterEncodingDetector
extends Object
implements org.apache.tika.detect.MetaEncodingDetector

A MetaEncodingDetector that arbitrates charset candidates by asking a TextQualityDetector which decoded candidate looks most like natural text.

Each base EncodingDetector in the CompositeEncodingDetector chain emits candidates into the EncodingDetectorContext. This meta detector then reads the raw probe bytes, decodes them under each unique candidate charset, scores each decode with TextQualityDetector.score(java.lang.String), and picks the highest-scoring candidate (the score is cross-script comparable, so a plain argmax suffices). BOM-declared, meta-tag-declared, structural, and statistical candidates all compete on the same footing — quality of the resulting decode is the sole criterion at this layer.

The TextQualityDetector implementation is discovered via ServiceLoader. When no implementation is on the classpath, this detector becomes a no-op: detect(org.apache.tika.io.TikaInputStream, org.apache.tika.metadata.Metadata, org.apache.tika.parser.ParseContext) returns an empty list and CompositeEncodingDetector falls back to its default confidence-based ordering.

Since:

Apache Tika 4.0.0 (TIKA-4720)

See Also:

• Serialized Form

Constructor Summary

Constructors

Constructor	Description
JunkFilterEncodingDetector()
JunkFilterEncodingDetector(org.apache.tika.quality.TextQualityDetector qualityDetector)	Test-only / deterministic-wiring constructor.

Method Summary

Modifier and Type	Method	Description
List<org.apache.tika.detect.EncodingResult>	detect(org.apache.tika.io.TikaInputStream tis, org.apache.tika.metadata.Metadata metadata, org.apache.tika.parser.ParseContext parseContext)
int	getReadLimit()
void	setReadLimit(int readLimit)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

JunkFilterEncodingDetector

public JunkFilterEncodingDetector()

JunkFilterEncodingDetector

public JunkFilterEncodingDetector(org.apache.tika.quality.TextQualityDetector qualityDetector)

Test-only / deterministic-wiring constructor.

Method Details

getReadLimit

public int getReadLimit()

setReadLimit

public void setReadLimit(int readLimit)

detect

public List<org.apache.tika.detect.EncodingResult> detect(org.apache.tika.io.TikaInputStream tis,
 org.apache.tika.metadata.Metadata metadata,
 org.apache.tika.parser.ParseContext parseContext)
                                                   throws IOException

Specified by:

detect in interface org.apache.tika.detect.EncodingDetector

Throws:

IOException