Package org.languagetool.tokenizers.en

Class EnglishWordTokenizer

java.lang.Object

org.languagetool.tokenizers.WordTokenizer

org.languagetool.tokenizers.en.EnglishWordTokenizer

All Implemented Interfaces:

Tokenizer

public class EnglishWordTokenizer
extends WordTokenizer

Since:

2.5

Author:

Marcin Milkowski

Field Summary

Fields inherited from class org.languagetool.tokenizers.WordTokenizer

REMOVED_EMOJI

Constructor Summary

Constructors

Constructor	Description
EnglishWordTokenizer()

Method Summary

Modifier and Type	Method	Description
List<String>	tokenize(String text)	Tokenizes text.

Methods inherited from class org.languagetool.tokenizers.WordTokenizer

getProtocols, getTokenizingCharacters, isCurrencyExpression, isEMail, isUrl, joinEMails, joinEMailsAndUrls, joinUrls, replaceEmojis, restoreEmojis, splitCurrencyExpression

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

EnglishWordTokenizer

public EnglishWordTokenizer()

Method Details

tokenize

public List<String> tokenize(String text)

Tokenizes text. The English tokenizer differs from the standard one in two respects:

1. it does not treat the hyphen as part of the word if the hyphen is at the end of the word;
2. it includes n-dash as a tokenizing character, as it is used without a whitespace in English.

Specified by:

tokenize in interface Tokenizer

Overrides:

tokenize in class WordTokenizer

Parameters:

text - String of words to tokenize.