Module igormaznitsa.prolog.parser

Package com.igormaznitsa.prologparser.tokenizer

Class Tokenizer

java.lang.Object

com.igormaznitsa.prologparser.tokenizer.Tokenizer

public final class Tokenizer
extends Object

Internal tokenizer to extract next token from reader.

Constructor Summary

Constructors

Constructor	Description
Tokenizer(PrologParser parser, Koi7CharOpMap metaOperators, Reader reader)

Method Summary

Modifier and Type	Method	Description
void	calcDiffAndPushResultBack(String etalon, StringBuilderEx buffer)
void	close(boolean closeReader)
OpContainer	findOperatorForName(String operatorName)
OpContainer	findOperatorForSingleChar(char c)
void	fixPosition()
int	getLastTokenLine()
int	getLastTokenPos()
int	getLine()
int	getPos()
boolean	hasOperatorStartsWith(String operatorNameStartSubstring)
static boolean	isCharAllowedForRadix(char chr, int radix)
PrologTerm	makeTermFromString(String str, int radix, Quotation quotingType, TokenizerState state)
TokenizerResult	peek()
TokenizerResult	pop()
void	push(char ch)
void	push(TokenizerResult object)
TokenizerResult	readNextToken()	Read next token

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Tokenizer

public Tokenizer(PrologParser parser,
                 Koi7CharOpMap metaOperators,
                 Reader reader)

Method Detail

isCharAllowedForRadix

public static boolean isCharAllowedForRadix(char chr,
                                            int radix)

calcDiffAndPushResultBack

public void calcDiffAndPushResultBack(String etalon,
                                      StringBuilderEx buffer)

push

public void push(char ch)

close

public void close(boolean closeReader)
           throws IOException

Throws:

IOException

hasOperatorStartsWith

public boolean hasOperatorStartsWith(String operatorNameStartSubstring)

findOperatorForName

public OpContainer findOperatorForName(String operatorName)

findOperatorForSingleChar

public OpContainer findOperatorForSingleChar(char c)

push

public void push(TokenizerResult object)

peek

public TokenizerResult peek()

getLastTokenPos

public int getLastTokenPos()

getLastTokenLine

public int getLastTokenLine()

fixPosition

public void fixPosition()

pop

public TokenizerResult pop()

readNextToken

public TokenizerResult readNextToken()

Read next token

Returns:

next token or null if not found or stream ended

makeTermFromString

public PrologTerm makeTermFromString(String str,
                                     int radix,
                                     Quotation quotingType,
                                     TokenizerState state)

getLine

public int getLine()

getPos

public int getPos()