Package io.karatelabs.parser

Class BaseLexer

java.lang.Object
io.karatelabs.parser.BaseLexer
Direct Known Subclasses:
JsLexer
public abstract class BaseLexer extends Object
Abstract base class for lexers. Provides common utilities for character handling, position tracking, and tokenization.

  • Field Details

    • resource

      protected final Resource resource

    • buffer

      protected final TokenBuffer buffer

    • source

      protected final String source

    • length

      protected final int length

    • pos

      protected int pos

    • line

      protected int line

    • col

      protected int col

    • tokenStart

      protected int tokenStart

    • tokenLine

      protected int tokenLine

    • tokenCol

      protected int tokenCol

  • Constructor Details

    • BaseLexer

      protected BaseLexer(Resource resource)

  • Method Details

    • nextToken

      public abstract Token nextToken()

    • scanToken

      protected abstract TokenType scanToken()

    • tokenize

      public static List<Token> tokenize(BaseLexer lexer)
      Tokenizes the source into a list of tokens. Tokens are stored in the lexer's buffer with prev/next navigation. Comments are collected for primary tokens.

    • isAtEnd

      protected boolean isAtEnd()

    • peek

      protected char peek()

    • peek

      protected char peek(int offset)

    • advance

      protected char advance()

    • match

      protected boolean match(char expected)

    • isDigit

      protected static boolean isDigit(char c)

    • isHexDigit

      protected static boolean isHexDigit(char c)

    • isIdentifierStart

      protected static boolean isIdentifierStart(char c)

    • isIdentifierPart

      protected static boolean isIdentifierPart(char c)