Package ai.djl.huggingface.tokenizers

Contains classes to interface with the underlying Huggingface tokenizers.

Class Summary

Class	Description
Encoding	A class holds token encoding information.
HuggingFaceTokenizer	HuggingFaceTokenizer is a Huggingface tokenizer implementation of the Tokenizer interface that converts sentences into token.
HuggingFaceTokenizer.Builder	The builder for creating huggingface tokenizer.