Package ai.djl.huggingface.tokenizers.jni

Contains classes to interface with the underlying Huggingface tokenizers.

Class Summary

Class	Description
CharSpan	A class holds character span information.
LibUtils	Utilities for finding the Huggingface tokenizer native binary on the System.
TokenizersLibrary	A class containing utilities to interact with the Tokenizer JNI layer.