public final class NLTokenUnit
extends java.lang.Object
NLTokenizer is a class used to automatically segment natural-language text. An instance of this class is created with a specific unit and assigned a string to tokenize, and clients can then obtain ranges for tokens in that string appropriate to the given unit. Units are defined by NLTokenUnit, which specifies the size of the units in a string to which tokenization or tagging applies, whether word, sentence, paragraph, or document.