Package ai.djl.huggingface.tokenizers

Class Encoding

  • public class Encoding
extends java.lang.Object
    A class holds token encoding information.

    • Constructor Summary

      Constructors 
      Modifier Constructor Description
      protected Encoding​(long[] ids, long[] typeIds, java.lang.String[] tokens, long[] wordIds, long[] attentionMask, long[] specialTokenMask, CharSpan[] charTokenSpans, Encoding[] overflowing)  

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      long[] getAttentionMask()
      Returns the attention masks.
      CharSpan[] getCharTokenSpans()
      Returns char token spans.
      long[] getIds()
      Returns the token ids.
      Encoding[] getOverflowing()
      Returns an array of overflowing encodings.
      long[] getSpecialTokenMask()
      Returns the special token masks.
      java.lang.String[] getTokens()
      Returns the tokens.
      long[] getTypeIds()
      Returns the token type ids.
      long[] getWordIds()
      Returns the word ids.
      ai.djl.ndarray.NDList toNDList​(ai.djl.ndarray.NDManager manager, boolean withTokenType)
      Returns the NDList representation of the encoding.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

      • Encoding

        protected Encoding​(long[] ids,
                   long[] typeIds,
                   java.lang.String[] tokens,
                   long[] wordIds,
                   long[] attentionMask,
                   long[] specialTokenMask,
                   CharSpan[] charTokenSpans,
                   Encoding[] overflowing)

    • Method Detail

      • toNDList

        public ai.djl.ndarray.NDList toNDList​(ai.djl.ndarray.NDManager manager,
                                      boolean withTokenType)
        Returns the NDList representation of the encoding.
        Parameters:
        manager - the NDManager to create the NDList
        withTokenType - true to include the token type id
        Returns:
        the NDList

      • getIds

        public long[] getIds()
        Returns the token ids.
        Returns:
        the token ids

      • getTypeIds

        public long[] getTypeIds()
        Returns the token type ids.
        Returns:
        the token type ids

      • getTokens

        public java.lang.String[] getTokens()
        Returns the tokens.
        Returns:
        the tokens

      • getWordIds

        public long[] getWordIds()
        Returns the word ids.
        Returns:
        the word ids

      • getAttentionMask

        public long[] getAttentionMask()
        Returns the attention masks.
        Returns:
        the attention masks

      • getSpecialTokenMask

        public long[] getSpecialTokenMask()
        Returns the special token masks.
        Returns:
        the special token masks

      • getCharTokenSpans

        public CharSpan[] getCharTokenSpans()
        Returns char token spans.
        Returns:
        char token spans

      • getOverflowing

        public Encoding[] getOverflowing()
        Returns an array of overflowing encodings.
        Returns:
        the array of overflowing encodings