Class Token

java.lang.Object
com.oracle.js.parser.Token
public final class Token extends Object
A token is a 64 bit long value that represents a basic parse/lex unit. This class provides static methods to manipulate lexer tokens.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final int
    LENGTH_MASK
    We use 28 bits for the position and 28 bits for the length of the token.

  • Method Summary

    Modifier and Type
    Method
    Description
    static int
    descLength(long token)
    Extract token length from a token descriptor.
    static int
    descPosition(long token)
    Extract token position from a token descriptor.
    static TokenType
    descType(long token)
    Extract token type from a token descriptor.
    static long
    recast(long token, TokenType newType)
    Change the token to use a new type.
    static long
    toDesc(TokenType type, int position, int length)
    Create a compact form of token information.
    static String
    toString(Source source, long token)
    String conversion of token
    static String
    toString(Source source, long token, boolean verbose)
    Return a string representation of a token.
    static long
    withDelimiter(long token)
    Normally returns the token itself, except in case of string tokens which report their position past their opening delimiter and thus need to have position and length adjusted.

    Methods inherited from class Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • LENGTH_MASK

      public static final int LENGTH_MASK
      We use 28 bits for the position and 28 bits for the length of the token. This limits the maximal length of code we can handle to 2 ^ 28 - 1 bytes.
      See Also:

  • Method Details

    • toDesc

      public static long toDesc(TokenType type, int position, int length)
      Create a compact form of token information.
      Parameters:
      type - Type of token.
      position - Start position of the token in the source.
      length - Length of the token.
      Returns:
      Token descriptor.

    • descPosition

      public static int descPosition(long token)
      Extract token position from a token descriptor.
      Parameters:
      token - Token descriptor.
      Returns:
      Start position of the token in the source.

    • withDelimiter

      public static long withDelimiter(long token)
      Normally returns the token itself, except in case of string tokens which report their position past their opening delimiter and thus need to have position and length adjusted.
      Parameters:
      token - Token descriptor.
      Returns:
      same or adjusted token.

    • descLength

      public static int descLength(long token)
      Extract token length from a token descriptor.
      Parameters:
      token - Token descriptor.
      Returns:
      Length of the token.

    • descType

      public static TokenType descType(long token)
      Extract token type from a token descriptor.
      Parameters:
      token - Token descriptor.
      Returns:
      Type of token.

    • recast

      public static long recast(long token, TokenType newType)
      Change the token to use a new type.
      Parameters:
      token - The original token.
      newType - The new token type.
      Returns:
      The recast token.

    • toString

      public static String toString(Source source, long token, boolean verbose)
      Return a string representation of a token.
      Parameters:
      source - Token source.
      token - Token descriptor.
      verbose - True to include details.
      Returns:
      String representation.

    • toString

      public static String toString(Source source, long token)
      String conversion of token
      Parameters:
      source - the source
      token - the token
      Returns:
      token as string