Package ai.djl.huggingface.tokenizers

Class HuggingFaceTokenizer

  • All Implemented Interfaces:
    ai.djl.modality.nlp.preprocess.TextProcessor, ai.djl.modality.nlp.preprocess.Tokenizer, java.lang.AutoCloseable
    public final class HuggingFaceTokenizer
extends ai.djl.util.NativeResource<java.lang.Long>
implements ai.djl.modality.nlp.preprocess.Tokenizer
    HuggingFaceTokenizer is a Huggingface tokenizer implementation of the Tokenizer interface that converts sentences into token.

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
      static class  HuggingFaceTokenizer.Builder
      The builder for creating huggingface tokenizer.

    • Field Summary

      • Fields inherited from class ai.djl.util.NativeResource

        handle

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      java.lang.String[] batchDecode​(long[][] batchIds)
      Returns the decoded Strings from the input batch ids.
      java.lang.String[] batchDecode​(long[][] batchIds, boolean skipSpecialTokens)
      Returns the decoded Strings from the input batch ids.
      Encoding[] batchEncode​(ai.djl.util.PairList<java.lang.String,​java.lang.String> inputs)
      Returns the Encoding of the input text pair in batch.
      Encoding[] batchEncode​(ai.djl.util.PairList<java.lang.String,​java.lang.String> inputs, boolean addSpecialTokens)
      Returns the Encoding of the input text pair in batch.
      Encoding[] batchEncode​(java.lang.String[] inputs)
      Returns the Encoding of the input sentence in batch.
      Encoding[] batchEncode​(java.lang.String[] inputs, boolean addSpecialTokens)
      Returns the Encoding of the input sentence in batch.
      Encoding[] batchEncode​(java.util.List<java.lang.String> inputs)
      Returns the Encoding of the input sentence in batch.
      Encoding[] batchEncode​(java.util.List<java.lang.String> inputs, boolean addSpecialTokens)
      Returns the Encoding of the input sentence in batch.
      static HuggingFaceTokenizer.Builder builder()
      Creates a builder to build a HuggingFaceTokenizer.
      static HuggingFaceTokenizer.Builder builder​(java.util.Map<java.lang.String,​?> arguments)
      Creates a builder to build a HuggingFaceTokenizer.
      java.lang.String buildSentence​(java.util.List<java.lang.String> tokens)
      void close()
      java.lang.String decode​(long[] ids)
      Returns the decoded String from the input ids.
      java.lang.String decode​(long[] ids, boolean skipSpecialTokens)
      Returns the decoded String from the input ids.
      void enableBatch()
      Sets padding and truncation to true for batching.
      Encoding encode​(java.lang.String text)
      Returns the Encoding of the input sentence.
      Encoding encode​(java.lang.String[] inputs)
      Returns the Encoding of the input sentences.
      Encoding encode​(java.lang.String[] inputs, boolean addSpecialTokens)
      Returns the Encoding of the input sentences.
      Encoding encode​(java.lang.String text, boolean addSpecialTokens)
      Returns the Encoding of the input sentence.
      Encoding encode​(java.lang.String text, java.lang.String textPair)
      Returns the Encoding of the input sentence.
      Encoding encode​(java.lang.String text, java.lang.String textPair, boolean addSpecialTokens)
      Returns the Encoding of the input sentence.
      Encoding encode​(java.util.List<java.lang.String> inputs)
      Returns the Encoding of the input sentences.
      Encoding encode​(java.util.List<java.lang.String> inputs, boolean addSpecialTokens)
      Returns the Encoding of the input sentences.
      protected void finalize()
      static HuggingFaceTokenizer newInstance​(java.io.InputStream is, java.util.Map<java.lang.String,​java.lang.String> options)
      Create a pre-trained HuggingFaceTokenizer instance from InputStream.
      static HuggingFaceTokenizer newInstance​(java.lang.String name)
      Creates a pre-trained HuggingFaceTokenizer instance from huggingface hub.
      static HuggingFaceTokenizer newInstance​(java.lang.String identifier, java.util.Map<java.lang.String,​java.lang.String> options)
      Create a pre-trained HuggingFaceTokenizer instance from huggingface hub.
      static HuggingFaceTokenizer newInstance​(java.nio.file.Path modelPath)
      Create a pre-trained HuggingFaceTokenizer instance from existing models.
      static HuggingFaceTokenizer newInstance​(java.nio.file.Path vocab, java.nio.file.Path merges, java.util.Map<java.lang.String,​java.lang.String> options)
      Create a pre-trained BPE HuggingFaceTokenizer instance from existing models.
      static HuggingFaceTokenizer newInstance​(java.nio.file.Path modelPath, java.util.Map<java.lang.String,​java.lang.String> options)
      Create a pre-trained HuggingFaceTokenizer instance from existing models.
      java.util.List<java.lang.String> tokenize​(java.lang.String sentence)

      • Methods inherited from class ai.djl.util.NativeResource

        getHandle, getUid, isReleased, onClose

      • Methods inherited from class java.lang.Object

        clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

      • Methods inherited from interface ai.djl.modality.nlp.preprocess.Tokenizer

        preprocess

    • Method Detail

      • newInstance

        public static HuggingFaceTokenizer newInstance​(java.lang.String name)
        Creates a pre-trained HuggingFaceTokenizer instance from huggingface hub.
        Parameters:
        name - the name of the huggingface tokenizer
        Returns:
        a HuggingFaceTokenizer instance

      • newInstance

        public static HuggingFaceTokenizer newInstance​(java.lang.String identifier,
                                               java.util.Map<java.lang.String,​java.lang.String> options)
        Create a pre-trained HuggingFaceTokenizer instance from huggingface hub.
        Parameters:
        identifier - the identifier of the huggingface tokenizer
        options - tokenizer options
        Returns:
        a HuggingFaceTokenizer instance

      • newInstance

        public static HuggingFaceTokenizer newInstance​(java.nio.file.Path modelPath)
                                        throws java.io.IOException
        Create a pre-trained HuggingFaceTokenizer instance from existing models.
        Parameters:
        modelPath - the directory or file path of the model location
        Returns:
        a HuggingFaceTokenizer instance
        Throws:
        java.io.IOException - when IO operation fails in loading a resource

      • newInstance

        public static HuggingFaceTokenizer newInstance​(java.nio.file.Path modelPath,
                                               java.util.Map<java.lang.String,​java.lang.String> options)
                                        throws java.io.IOException
        Create a pre-trained HuggingFaceTokenizer instance from existing models.
        Parameters:
        modelPath - the directory or file path of the model location
        options - tokenizer options
        Returns:
        a HuggingFaceTokenizer instance
        Throws:
        java.io.IOException - when IO operation fails in loading a resource

      • newInstance

        public static HuggingFaceTokenizer newInstance​(java.nio.file.Path vocab,
                                               java.nio.file.Path merges,
                                               java.util.Map<java.lang.String,​java.lang.String> options)
                                        throws java.io.IOException
        Create a pre-trained BPE HuggingFaceTokenizer instance from existing models.
        Parameters:
        vocab - the BPE vocabulary file
        merges - the BPE merges file
        options - tokenizer options
        Returns:
        a HuggingFaceTokenizer instance
        Throws:
        java.io.IOException - when IO operation fails in loading a resource

      • newInstance

        public static HuggingFaceTokenizer newInstance​(java.io.InputStream is,
                                               java.util.Map<java.lang.String,​java.lang.String> options)
                                        throws java.io.IOException
        Create a pre-trained HuggingFaceTokenizer instance from InputStream.
        Parameters:
        is - InputStream
        options - tokenizer options
        Returns:
        a HuggingFaceTokenizer instance
        Throws:
        java.io.IOException - when IO operation fails in loading a resource

      • tokenize

        public java.util.List<java.lang.String> tokenize​(java.lang.String sentence)
        Specified by:
        tokenize in interface ai.djl.modality.nlp.preprocess.Tokenizer

      • buildSentence

        public java.lang.String buildSentence​(java.util.List<java.lang.String> tokens)
        Specified by:
        buildSentence in interface ai.djl.modality.nlp.preprocess.Tokenizer

      • close

        public void close()
        Specified by:
        close in interface java.lang.AutoCloseable
        Overrides:
        close in class ai.djl.util.NativeResource<java.lang.Long>

      • encode

        public Encoding encode​(java.lang.String text,
                       boolean addSpecialTokens)
        Returns the Encoding of the input sentence.
        Parameters:
        text - the input sentence
        addSpecialTokens - whether to encode the sequence with special tokens relative to their model
        Returns:
        the Encoding of the input sentence

      • encode

        public Encoding encode​(java.lang.String text)
        Returns the Encoding of the input sentence.
        Parameters:
        text - the input sentence
        Returns:
        the Encoding of the input sentence

      • encode

        public Encoding encode​(java.lang.String text,
                       java.lang.String textPair,
                       boolean addSpecialTokens)
        Returns the Encoding of the input sentence.
        Parameters:
        text - the input sentence
        textPair - the second input sentence
        addSpecialTokens - whether to encode the sequence with special tokens relative to their model
        Returns:
        the Encoding of the input sentence

      • encode

        public Encoding encode​(java.lang.String text,
                       java.lang.String textPair)
        Returns the Encoding of the input sentence.
        Parameters:
        text - the input sentence
        textPair - the second input sentence
        Returns:
        the Encoding of the input sentence

      • encode

        public Encoding encode​(java.util.List<java.lang.String> inputs,
                       boolean addSpecialTokens)
        Returns the Encoding of the input sentences.
        Parameters:
        inputs - the input sentences
        addSpecialTokens - whether to encode the sequence with special tokens relative to their model
        Returns:
        the Encoding of the input sentences

      • encode

        public Encoding encode​(java.util.List<java.lang.String> inputs)
        Returns the Encoding of the input sentences.
        Parameters:
        inputs - the input sentences
        Returns:
        the Encoding of the input sentences

      • encode

        public Encoding encode​(java.lang.String[] inputs,
                       boolean addSpecialTokens)
        Returns the Encoding of the input sentences.
        Parameters:
        inputs - the input sentences
        addSpecialTokens - whether to encode the sequence with special tokens relative to their model
        Returns:
        the Encoding of the input sentences

      • encode

        public Encoding encode​(java.lang.String[] inputs)
        Returns the Encoding of the input sentences.
        Parameters:
        inputs - the input sentences
        Returns:
        the Encoding of the input sentences

      • batchEncode

        public Encoding[] batchEncode​(java.util.List<java.lang.String> inputs,
                              boolean addSpecialTokens)
        Returns the Encoding of the input sentence in batch.
        Parameters:
        inputs - the batch of input sentence
        addSpecialTokens - whether to encode the sequence with special tokens relative to their model
        Returns:
        the Encoding of the input sentence in batch

      • batchEncode

        public Encoding[] batchEncode​(java.util.List<java.lang.String> inputs)
        Returns the Encoding of the input sentence in batch.
        Parameters:
        inputs - the batch of input sentence
        Returns:
        the Encoding of the input sentence in batch

      • batchEncode

        public Encoding[] batchEncode​(java.lang.String[] inputs,
                              boolean addSpecialTokens)
        Returns the Encoding of the input sentence in batch.
        Parameters:
        inputs - the batch of input sentence
        addSpecialTokens - whether to encode the sequence with special tokens relative to their model
        Returns:
        the Encoding of the input sentence in batch

      • batchEncode

        public Encoding[] batchEncode​(java.lang.String[] inputs)
        Returns the Encoding of the input sentence in batch.
        Parameters:
        inputs - the batch of input sentence
        Returns:
        the Encoding of the input sentence in batch

      • batchEncode

        public Encoding[] batchEncode​(ai.djl.util.PairList<java.lang.String,​java.lang.String> inputs,
                              boolean addSpecialTokens)
        Returns the Encoding of the input text pair in batch.
        Parameters:
        inputs - the batch of input text pair
        addSpecialTokens - whether to encode the sequence with special tokens relative to their model
        Returns:
        the Encoding of the input text pair in batch

      • batchEncode

        public Encoding[] batchEncode​(ai.djl.util.PairList<java.lang.String,​java.lang.String> inputs)
        Returns the Encoding of the input text pair in batch.
        Parameters:
        inputs - the batch of input text pair
        Returns:
        the Encoding of the input text pair in batch

      • decode

        public java.lang.String decode​(long[] ids,
                               boolean skipSpecialTokens)
        Returns the decoded String from the input ids.
        Parameters:
        ids - the input ids
        skipSpecialTokens - whether to remove special tokens in the decoding
        Returns:
        the decoded String from the input ids

      • decode

        public java.lang.String decode​(long[] ids)
        Returns the decoded String from the input ids.
        Parameters:
        ids - the input ids
        Returns:
        the decoded String from the input ids

      • batchDecode

        public java.lang.String[] batchDecode​(long[][] batchIds,
                                      boolean skipSpecialTokens)
        Returns the decoded Strings from the input batch ids.
        Parameters:
        batchIds - the batch of id sequences to decode
        skipSpecialTokens - whether to remove special tokens in the decoding
        Returns:
        the decoded Strings from the input batch ids

      • batchDecode

        public java.lang.String[] batchDecode​(long[][] batchIds)
        Returns the decoded Strings from the input batch ids.
        Parameters:
        batchIds - the batch of id sequences to decode
        Returns:
        the decoded Strings from the input batch ids

      • enableBatch

        public void enableBatch()
        Sets padding and truncation to true for batching.

      • builder

        public static HuggingFaceTokenizer.Builder builder​(java.util.Map<java.lang.String,​?> arguments)
        Creates a builder to build a HuggingFaceTokenizer.
        Parameters:
        arguments - the models' arguments
        Returns:
        a new builder

      • finalize

        protected void finalize()
                 throws java.lang.Throwable
        Overrides:
        finalize in class java.lang.Object
        Throws:
        java.lang.Throwable