org.carrot2.text.preprocessing

Class Tokenizer

java.lang.Object

org.carrot2.text.preprocessing.Tokenizer

@Bindable(prefix="Tokenizer")
public final class Tokenizer
extends Object

Performs tokenization of documents.

This class saves the following results to the PreprocessingContext:

• PreprocessingContext.AllTokens.image
• PreprocessingContext.AllTokens.documentIndex
• PreprocessingContext.AllTokens.fieldIndex
• PreprocessingContext.AllTokens.type

Field Summary

Fields

Modifier and Type	Field and Description
Collection<String>	documentFields Textual fields of documents that should be tokenized and parsed for clustering.

Constructor Summary

Constructors

Constructor and Description
Tokenizer()

Method Summary

Methods

Modifier and Type	Method and Description
void	tokenize(PreprocessingContext context) Performs tokenization and saves the results to the context.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

documentFields

@Input
@Attribute
@Label(value="Document fields")
@Level(value=ADVANCED)
@Group(value="Preprocessing")
public Collection<String> documentFields

Textual fields of documents that should be tokenized and parsed for clustering.

Constructor Detail

Tokenizer

public Tokenizer()

Method Detail

tokenize

public void tokenize(PreprocessingContext context)

Performs tokenization and saves the results to the context.