Package ai.djl.huggingface.tokenizers

Class HuggingFaceTokenizer.Builder

  • Enclosing class:
    HuggingFaceTokenizer
    public static final class HuggingFaceTokenizer.Builder
extends java.lang.Object
    The builder for creating huggingface tokenizer.

    • Method Detail

      • optManager

        public HuggingFaceTokenizer.Builder optManager​(ai.djl.ndarray.NDManager manager)
        Sets the optional manager used to manage the lifecycle of the tokenizer.
        Parameters:
        manager - the NDManager
        Returns:
        this builder

      • optTokenizerName

        public HuggingFaceTokenizer.Builder optTokenizerName​(java.lang.String tokenizerName)
        Sets the name of the tokenizer.
        Parameters:
        tokenizerName - the name of the tokenizer
        Returns:
        this builder

      • optTokenizerPath

        public HuggingFaceTokenizer.Builder optTokenizerPath​(java.nio.file.Path tokenizerPath)
        Sets the file path of the tokenizer.
        Parameters:
        tokenizerPath - the path of the tokenizer
        Returns:
        this builder

      • optAddSpecialTokens

        public HuggingFaceTokenizer.Builder optAddSpecialTokens​(boolean addSpecialTokens)
        Sets if add special tokens.
        Parameters:
        addSpecialTokens - true to add special tokens
        Returns:
        this builder

      • optTruncation

        public HuggingFaceTokenizer.Builder optTruncation​(boolean enabled)
        Enables or Disables default truncation behavior for the tokenizer.
        Parameters:
        enabled - whether to enable default truncation behavior
        Returns:
        this builder

      • optTruncateFirstOnly

        public HuggingFaceTokenizer.Builder optTruncateFirstOnly()
        Enables truncation to only truncate the first item.
        Returns:
        this builder

      • optTruncateSecondOnly

        public HuggingFaceTokenizer.Builder optTruncateSecondOnly()
        Enables truncation to only truncate the second item.
        Returns:
        this builder

      • optPadding

        public HuggingFaceTokenizer.Builder optPadding​(boolean enabled)
        Enables or Disables default padding behavior for the tokenizer.
        Parameters:
        enabled - whether to enable default padding behavior
        Returns:
        this builder

      • optPadToMaxLength

        public HuggingFaceTokenizer.Builder optPadToMaxLength()
        Enables padding to pad sequences to previously specified maxLength, or modelMaxLength if not specified.
        Returns:
        this builder

      • optMaxLength

        public HuggingFaceTokenizer.Builder optMaxLength​(int maxLength)
        Sets maxLength for padding and truncation.
        Parameters:
        maxLength - the length to truncate and/or pad sequences to
        Returns:
        this builder

      • optPadToMultipleOf

        public HuggingFaceTokenizer.Builder optPadToMultipleOf​(int padToMultipleOf)
        Sets padToMultipleOf for padding.
        Parameters:
        padToMultipleOf - the multiple of sequences should be padded to
        Returns:
        this builder

      • optStride

        public HuggingFaceTokenizer.Builder optStride​(int stride)
        Sets the stride to use in overflow overlap when truncating sequences longer than the model supports.
        Parameters:
        stride - the number of tokens to overlap when truncating long sequences
        Returns:
        this builder

      • configure

        public void configure​(java.util.Map<java.lang.String,​?> arguments)
        Configures the builder with the arguments.
        Parameters:
        arguments - the arguments

      • build

        public HuggingFaceTokenizer build()
                           throws java.io.IOException
        Builds the translator.
        Returns:
        the new translator
        Throws:
        java.io.IOException - when IO operation fails in loading a resource