Index (Apache Tika ML junk detector — runtime and training tools 4.0.0 API)

A B C D E F G H J K L N O P R S T U V W Z
All Classes and Interfaces|All Packages|Constant Field Values|Serialized Form

A

accept(String, int, int) - Method in interface org.apache.tika.ml.junkdetect.JunkDetector.BigramSink
ALL_COMMON - Enum constant in enum class org.apache.tika.ml.junkdetect.TextQualityFeatures.StripMode: Production behaviour today: strip every COMMON, INHERITED, and UNKNOWN-script codepoint.
alphabeticRatio(String) - Static method in class org.apache.tika.ml.junkdetect.TextQualityFeatures: z6: fraction of codepoints that are letters (Character.isLetter(int)).

B

bigramSlots() - Method in class org.apache.tika.ml.junkdetect.BigramTables: Number of bigram-table slots (capacity).
BigramTables - Class in org.apache.tika.ml.junkdetect: Carrier for one script's codepoint-bigram tables.
BigramTables(int[], int[], byte[], byte[], float, float, float, float, float, float) - Constructor for class org.apache.tika.ml.junkdetect.BigramTables
bucketCount() - Static method in class org.apache.tika.ml.junkdetect.UnicodeBlockRanges: Total number of buckets (named blocks + 1 unassigned).
bucketOf(int) - Static method in class org.apache.tika.ml.junkdetect.UnicodeBlockRanges: Returns the bucket id for the given codepoint, or UnicodeBlockRanges.UNASSIGNED if the codepoint falls outside every named block range.
bucketSumsAndCounts(int[], Map<String, BigramTables>) - Static method in class org.apache.tika.ml.junkdetect.JunkDetector: Per-script {sumLogP, count} buckets for a codepoint sequence, scored with the given per-script tables.

C

classifierWeights - Variable in class org.apache.tika.ml.junkdetect.JunkDetector.FeatureComponents
clean(String) - Static method in class org.apache.tika.ml.junkdetect.HtmlContentCleaner: Strip HTML tags (entities preserved through the strip), then expand entities to real codepoints.
codepointCount() - Method in class org.apache.tika.ml.junkdetect.BigramTables: Number of codepoints in this script's index.
codepointToIndex(BigramTables, int) - Static method in class org.apache.tika.ml.junkdetect.JunkDetector: Binary-search a codepoint in the script's index.
combiningMarkRatio(String) - Static method in class org.apache.tika.ml.junkdetect.TextQualityFeatures: z10: fraction of codepoints that are combining or spacing marks (Unicode general categories Mn, Mc, Me).
COMMON_SCRIPT - Static variable in class org.apache.tika.ml.junkdetect.JunkDetector: Model script key for the pooled COMMON (digits/punctuation/symbols) table.
compare(String, String, String, String) - Method in class org.apache.tika.ml.junkdetect.JunkDetector
computeF1MeanLogP(int[], BigramTables) - Static method in class org.apache.tika.ml.junkdetect.JunkDetector: Codepoint-array form of JunkDetector.computeF1MeanLogP(String, BigramTables).
computeF1MeanLogP(String, BigramTables) - Static method in class org.apache.tika.ml.junkdetect.JunkDetector: Mean log-prob over the codepoint pairs in text using the given script's bigram tables.
computeZ2BlockTransition(String, float[], float[]) - Static method in class org.apache.tika.ml.junkdetect.JunkDetector: Feature 2 — calibrated z-score for block-transition mean log-prob on one text window.
computeZ3ControlByte(byte[], float[]) - Static method in class org.apache.tika.ml.junkdetect.JunkDetector: Feature 3 — calibrated z-score for control-byte fraction on the UTF-8 byte sequence of one text window.
computeZ4ScriptTransition(String, float[], float[], Map<String, Integer>, int) - Static method in class org.apache.tika.ml.junkdetect.JunkDetector: Feature 4 — calibrated z-score for global script-transition mean log-prob on one text window.
computeZ5LetterAdjacentToMarkRatio(String) - Method in class org.apache.tika.ml.junkdetect.JunkDetector: z5: calibrated letter-adjacent-to-mark ratio.
computeZ6ReplacementRatio(String) - Method in class org.apache.tika.ml.junkdetect.JunkDetector: z6: calibrated replacement-character ratio.
computeZ9AlternationRatio(String) - Method in class org.apache.tika.ml.junkdetect.JunkDetector: z9: calibrated script-alternation ratio.

D

DEFAULT_MODEL_RESOURCE - Static variable in class org.apache.tika.ml.junkdetect.JunkDetector: Classpath resource path for the bundled production model.
detect(TikaInputStream, Metadata, ParseContext) - Method in class org.apache.tika.ml.junkdetect.JunkFilterEncodingDetector
dominantScript - Variable in class org.apache.tika.ml.junkdetect.JunkDetector.FeatureComponents

E

EMPTY_KEY - Static variable in class org.apache.tika.ml.junkdetect.BigramTables: Reserved value in BigramTables.bigramKeys marking an unoccupied slot.

F

forEachScriptBigram(int[], JunkDetector.BigramSink) - Static method in class org.apache.tika.ml.junkdetect.JunkDetector

G

getModelVersion() - Method in class org.apache.tika.ml.junkdetect.JunkDetector: Returns the file-format version of the loaded model (always JunkDetector.VERSION; mismatches are rejected at load time).
getReadLimit() - Method in class org.apache.tika.ml.junkdetect.JunkFilterEncodingDetector

H

highByteEntropy(String) - Static method in class org.apache.tika.ml.junkdetect.TextQualityFeatures: z7: Shannon entropy (in bits) of the distribution of distinct codepoints in the high-byte range (cp >= 0x80).
HtmlContentCleaner - Class in org.apache.tika.ml.junkdetect: Junk-detection HTML→text cleaning: strip tags, then expand entities to codepoints.

J

JunkDetector - Class in org.apache.tika.ml.junkdetect: Language-agnostic text quality scorer.
JunkDetector.BigramSink - Interface in org.apache.tika.ml.junkdetect: Sink for JunkDetector.forEachScriptBigram(int[], org.apache.tika.ml.junkdetect.JunkDetector.BigramSink): (modelScript, cpA, cpB).
JunkDetector.FeatureComponents - Class in org.apache.tika.ml.junkdetect: Per-feature z-score breakdown returned by JunkDetector.scoreWithFeatureComponents(String).
JunkFilterEncodingDetector - Class in org.apache.tika.ml.junkdetect: A MetaEncodingDetector that arbitrates charset candidates by asking a TextQualityDetector which decoded candidate looks most like natural text.
JunkFilterEncodingDetector() - Constructor for class org.apache.tika.ml.junkdetect.JunkFilterEncodingDetector
JunkFilterEncodingDetector(TextQualityDetector) - Constructor for class org.apache.tika.ml.junkdetect.JunkFilterEncodingDetector: Test-only / deterministic-wiring constructor.

K

knownScripts() - Method in class org.apache.tika.ml.junkdetect.JunkDetector: Returns the set of script names this model knows about.

L

letterAdjacentToMarkRatio(String) - Static method in class org.apache.tika.ml.junkdetect.TextQualityFeatures: z11: fraction of adjacent codepoint pairs where the first is a letter and the second is a combining or spacing mark.
letterPairDensity(String) - Static method in class org.apache.tika.ml.junkdetect.TextQualityFeatures: z5: fraction of adjacent codepoint pairs where both codepoints are letters in the same script cluster.
load(InputStream) - Static method in class org.apache.tika.ml.junkdetect.JunkDetector: Loads a model from an InputStream.
loadFromClasspath() - Static method in class org.apache.tika.ml.junkdetect.JunkDetector: Loads the bundled model from the classpath.
loadFromPath(Path) - Static method in class org.apache.tika.ml.junkdetect.JunkDetector: Loads a model from the given file path.
logit - Variable in class org.apache.tika.ml.junkdetect.JunkDetector.FeatureComponents

N

NONE - Enum constant in enum class org.apache.tika.ml.junkdetect.TextQualityFeatures.StripMode: No stripping — pass text through unchanged.

O

org.apache.tika.ml.junkdetect - package org.apache.tika.ml.junkdetect

P

packBigramKey(int, int) - Static method in class org.apache.tika.ml.junkdetect.JunkDetector: Packed bigram key for indices (a, b) where each index fits in JunkDetectorTrainingConfig#KEY_INDEX_BITS bits.
perWordScriptPurity(String) - Static method in class org.apache.tika.ml.junkdetect.TextQualityFeatures: Candidate feature (not currently in the classifier): fraction of whitespace-delimited tokens whose letter codepoints all belong to the same script cluster.
provider() - Static method in class org.apache.tika.ml.junkdetect.JunkDetector: ServiceLoader provider hook (Java 9+).

R

readFrom(DataInputStream) - Static method in class org.apache.tika.ml.junkdetect.BigramTables: Inverse of BigramTables.writeTo(DataOutputStream).
replacementCount(String) - Static method in class org.apache.tika.ml.junkdetect.TextQualityFeatures: Raw count of U+FFFD codepoints.
replacementRatio(String) - Static method in class org.apache.tika.ml.junkdetect.TextQualityFeatures: z6 raw input: fraction of codepoints that are "anomaly indicators" — codepoints that shouldn't appear in correctly-decoded natural text.

S

SCHEME_VERSION - Static variable in class org.apache.tika.ml.junkdetect.UnicodeBlockRanges: Bumped whenever the static range table below changes.
score(String) - Method in class org.apache.tika.ml.junkdetect.JunkDetector
scoreWithFeatureComponents(String) - Method in class org.apache.tika.ml.junkdetect.JunkDetector: Diagnostic — exposes the per-feature z-scores, the global combiner weights, and the dominant script.
scriptAlternationRatio(String) - Static method in class org.apache.tika.ml.junkdetect.TextQualityFeatures: z9: script-alternation ratio — observed transitions over expected transitions under a random-shuffle null.
scriptDensity(String) - Static method in class org.apache.tika.ml.junkdetect.TextQualityFeatures: Fraction of codepoints assigned to a "real" script (i.e. not in COMMON / INHERITED / UNKNOWN).
scriptFragmentation(String) - Static method in class org.apache.tika.ml.junkdetect.TextQualityFeatures: Fragmentation of script-bearing codepoints across distinct scripts: 1 - longest_same_script_run_length / total_script_codepoints.
setReadLimit(int) - Method in class org.apache.tika.ml.junkdetect.JunkFilterEncodingDetector
statsString() - Method in class org.apache.tika.ml.junkdetect.BigramTables: Returns a one-line summary for trainer progress output.
strip(String, TextQualityFeatures.StripMode) - Static method in class org.apache.tika.ml.junkdetect.TextQualityFeatures

T

TextQualityFeatures - Class in org.apache.tika.ml.junkdetect: Candidate charset-agnostic features for use in the Phase-2 feature study (see plan wild-roaming-whale).
TextQualityFeatures.StripMode - Enum Class in org.apache.tika.ml.junkdetect
TOKEN_END - Static variable in class org.apache.tika.ml.junkdetect.JunkDetector
TOKEN_START - Static variable in class org.apache.tika.ml.junkdetect.JunkDetector: Sentinel "codepoints" (above the Unicode maximum U+10FFFF, so they cannot collide with real text) that wrap each letter token: TOKEN_START (^) before the first letter of a run and TOKEN_END ($) after the last.
totalBytes - Variable in class org.apache.tika.ml.junkdetect.JunkDetector.FeatureComponents

U

UNASSIGNED - Static variable in class org.apache.tika.ml.junkdetect.UnicodeBlockRanges: Bucket index returned for codepoints in no named block.
UnicodeBlockRanges - Class in org.apache.tika.ml.junkdetect: Static codepoint-range → bucket-index lookup table used by Feature 2 (block-transition log-probability).

V

valueOf(String) - Static method in enum class org.apache.tika.ml.junkdetect.TextQualityFeatures.StripMode: Returns the enum constant of this class with the specified name.
values() - Static method in enum class org.apache.tika.ml.junkdetect.TextQualityFeatures.StripMode: Returns an array containing the constants of this enum class, in the order they are declared.
VERSION - Static variable in class org.apache.tika.ml.junkdetect.JunkDetector: Sole supported file-format version.

W

WHITESPACE - Enum constant in enum class org.apache.tika.ml.junkdetect.TextQualityFeatures.StripMode: Strip only Unicode whitespace (per Character.isWhitespace(int)).
WHITESPACE_CONTROL - Enum constant in enum class org.apache.tika.ml.junkdetect.TextQualityFeatures.StripMode: Strip whitespace plus control characters and format characters (general categories Cc, Cf).
writeTo(DataOutputStream) - Method in class org.apache.tika.ml.junkdetect.BigramTables: Serialises this script's bigram tables.

Z

z1 - Variable in class org.apache.tika.ml.junkdetect.JunkDetector.FeatureComponents
z2 - Variable in class org.apache.tika.ml.junkdetect.JunkDetector.FeatureComponents
z3 - Variable in class org.apache.tika.ml.junkdetect.JunkDetector.FeatureComponents
z4 - Variable in class org.apache.tika.ml.junkdetect.JunkDetector.FeatureComponents
z5 - Variable in class org.apache.tika.ml.junkdetect.JunkDetector.FeatureComponents
z6 - Variable in class org.apache.tika.ml.junkdetect.JunkDetector.FeatureComponents
z7 - Variable in class org.apache.tika.ml.junkdetect.JunkDetector.FeatureComponents
z8 - Variable in class org.apache.tika.ml.junkdetect.JunkDetector.FeatureComponents
z9 - Variable in class org.apache.tika.ml.junkdetect.JunkDetector.FeatureComponents

A B C D E F G H J K L N O P R S T U V W Z
All Classes and Interfaces|All Packages|Constant Field Values|Serialized Form