org.apache.flink.cdc.common.text

类 TokenStream

java.lang.Object

org.apache.flink.cdc.common.text.TokenStream

@NotThreadSafe
public class TokenStream
extends Object

A stream of tokens that can be consumed by a parser. This class is not thread-safe.

嵌套类概要

嵌套类

限定符和类型	类和说明
protected class	TokenStream.CaseInsensitiveToken An immutable TokenStream.Token that implements matching using case-insensitive logic.
class	TokenStream.CaseInsensitiveTokenFactory An implementation of TokenStream.Tokens that creates TokenStream.CaseInsensitiveToken objects.
protected class	TokenStream.CaseSensitiveToken An immutable TokenStream.Token that implements matching using case-sensitive logic.
class	TokenStream.CaseSensitiveTokenFactory An implementation of TokenStream.Tokens that creates TokenStream.CaseSensitiveToken objects.
static class	TokenStream.CharacterArrayStream An implementation of TokenStream.CharacterStream that works with a single character array.
static interface	TokenStream.CharacterStream Interface used by a TokenStream.Tokenizer to iterate through the characters in the content input to the TokenStream.
static class	TokenStream.Marker An opaque marker for a position within the token stream.
static interface	TokenStream.Token The interface defining a token, which references the characters in the actual input character stream.
protected class	TokenStream.TokenFactory An implementation of TokenStream.Tokens that creates TokenStream.CaseSensitiveToken objects.
static interface	TokenStream.Tokenizer Interface for a Tokenizer component responsible for processing the characters in a TokenStream.CharacterStream and constructing the appropriate TokenStream.Token objects.
static interface	TokenStream.Tokens A factory for Token objects, used by a TokenStream.Tokenizer to create tokens in the correct order.

字段概要

字段

限定符和类型	字段和说明
static int	ANY_TYPE
static String	ANY_VALUE
protected String	inputString

构造器概要

构造器

构造器和说明
TokenStream(String content, TokenStream.Tokenizer tokenizer, boolean caseSensitive)

方法概要

限定符和类型	方法和说明
String	consume() Return the value of this token and move to the next token.
boolean	hasNext() Determine if this stream has another token to be consumed.
protected List<TokenStream.Token>	initializeTokens(List<TokenStream.Token> tokens) Method to allow subclasses to pre-process the set of tokens and return the correct tokens to use.
TokenStream.Marker	mark() Obtain a marker that records the current position so that the stream can be rewind(Marker) back to the mark even after having been advanced beyond the mark.
boolean	matches(int type, String expected) Determine if the current token matches the expected type and a value.
boolean	matches(String expected) Determine if the current token matches the expected value.
boolean	matchesAnyOf(String[] options) Determine if the next token matches one of the supplied values.
String	peek()
boolean	rewind(TokenStream.Marker marker) Reset the stream back to the position described by the supplied marker.
TokenStream	start() Begin the token stream, including (if required) the tokenization of the input content.
protected void	throwNoMoreContent()
String	toString()

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

字段详细资料

ANY_VALUE

public static final String ANY_VALUE

另请参阅:

常量字段值

ANY_TYPE

public static final int ANY_TYPE

另请参阅:

常量字段值

inputString

protected final String inputString

构造器详细资料

TokenStream

public TokenStream(String content,
                   TokenStream.Tokenizer tokenizer,
                   boolean caseSensitive)

方法详细资料

start

public TokenStream start()
                  throws ParsingException

Begin the token stream, including (if required) the tokenization of the input content.

返回:

this object for easy method chaining; never null

抛出:

ParsingException - if an error occurs during tokenization of the content

initializeTokens

protected List<TokenStream.Token> initializeTokens(List<TokenStream.Token> tokens)

Method to allow subclasses to pre-process the set of tokens and return the correct tokens to use. The default behavior is to simply return the supplied tokens.

参数:

tokens - the tokens

返回:

list of tokens.

mark

public TokenStream.Marker mark()

Obtain a marker that records the current position so that the stream can be rewind(Marker) back to the mark even after having been advanced beyond the mark.

返回:

the marker; never null

抛出:

IllegalStateException - if this method was called before the stream was started

NoSuchElementException - if there are no more tokens

rewind

public boolean rewind(TokenStream.Marker marker)

Reset the stream back to the position described by the supplied marker. This method does nothing if the mark is invalid. For example, it is not possible to advance the token stream beyond the current position.

参数:

marker - the marker

返回:

true if the token stream was reset, or false if the marker was invalid

consume

public String consume()
               throws ParsingException,
                      IllegalStateException

Return the value of this token and move to the next token.

返回:

the value of the current token

抛出:

ParsingException - if there is no such token to consume

IllegalStateException - if this method was called before the stream was started

throwNoMoreContent

protected void throwNoMoreContent()
                           throws ParsingException

抛出:

ParsingException

peek

public String peek()
            throws IllegalStateException

抛出:

IllegalStateException

matches

public boolean matches(String expected)
                throws IllegalStateException

Determine if the current token matches the expected value.

The ANY_VALUE constant can be used as a wildcard.

参数:

expected - the expected value of the current token

返回:

true if the current token did match, or false if the current token did not match

抛出:

IllegalStateException - if this method was called before the stream was started

matches

public boolean matches(int type,
                       String expected)
                throws IllegalStateException

Determine if the current token matches the expected type and a value.

The ANY_VALUE constant can be used as a wildcard.

参数:

type - the expected type of the curent token

expected - the expected value of the current token

返回:

true if the current token did match, or false if the current token did not match

抛出:

IllegalStateException - if this method was called before the stream was started

matchesAnyOf

public boolean matchesAnyOf(String[] options)
                     throws IllegalStateException

Determine if the next token matches one of the supplied values.

参数:

options - the options for the value of the current token

返回:

true if the current token's value did match one of the supplied options, or false otherwise

抛出:

IllegalStateException - if this method was called before the stream was started

hasNext

public boolean hasNext()

Determine if this stream has another token to be consumed.

返回:

true if there is another token ready for consumption, or false otherwise

抛出:

IllegalStateException - if this method was called before the stream was started

toString

public String toString()

覆盖:

toString 在类中 Object