Package org.dizitart.no2.index.fulltext

Class UniversalTextTokenizer

java.lang.Object

org.dizitart.no2.index.fulltext.BaseTextTokenizer

org.dizitart.no2.index.fulltext.UniversalTextTokenizer

All Implemented Interfaces:

TextTokenizer

public class UniversalTextTokenizer
extends BaseTextTokenizer

A TextTokenizer implementation that tokenizes text using a universal approach.

Since:

2.1.0

Author:

Anindya Chatterjee

See Also:

• Languages

Constructor Summary

Constructors

Constructor	Description
UniversalTextTokenizer()
UniversalTextTokenizer(Languages... languages)

Method Summary

Modifier and Type	Method	Description
Languages	getLanguage()	Gets the language for the tokenizer.
Set<String>	stopWords()	Gets all stop-words for a language.

Methods inherited from class org.dizitart.no2.index.fulltext.BaseTextTokenizer

tokenize

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

UniversalTextTokenizer

public UniversalTextTokenizer()

UniversalTextTokenizer

public UniversalTextTokenizer(Languages... languages)

Method Details

getLanguage

public Languages getLanguage()

Description copied from interface: TextTokenizer

Gets the language for the tokenizer.

Returns:

the language for this tokenizer.

stopWords

public Set<String> stopWords()

Description copied from interface: TextTokenizer

Gets all stop-words for a language.

Returns:

the set of all stop-words.