Package org.dizitart.no2.index.fulltext

Interface TextTokenizer

All Known Implementing Classes:
BaseTextTokenizer, EnglishTextTokenizer, UniversalTextTokenizer
public interface TextTokenizer
An abstract class representing a stop-word based text tokenizer.
Since:
1.0
Author:
Anindya Chatterjee.
See Also:

  • Method Summary

    Modifier and Type
    Method
    Description
    Languages
    getLanguage()
    Gets the language for the tokenizer.
    Set<String>
    stopWords()
    Gets all stop-words for a language.
    Set<String>
    tokenize(String text)
    Tokenize a text and discards all stop-words from it.

  • Method Details

    • getLanguage

      Languages getLanguage()
      Gets the language for the tokenizer.
      Returns:
      the language for this tokenizer.

    • tokenize

      Set<String> tokenize(String text)
      Tokenize a text and discards all stop-words from it.
      Parameters:
      text - the text to tokenize
      Returns:
      the set of tokens.

    • stopWords

      Set<String> stopWords()
      Gets all stop-words for a language.
      Returns:
      the set of all stop-words.