com.metaweb.lessen.tokenizers
Class RegroupingTokenizer

java.lang.Object
  com.metaweb.lessen.tokenizers.RegroupingTokenizer

All Implemented Interfaces:

Tokenizer

public class RegroupingTokenizer
extends Object
implements Tokenizer

Field Summary

protected Token	_token
protected BufferedTokenizer	_tokenizer

Constructor Summary

RegroupingTokenizer(Tokenizer tokenizer)

Method Summary

protected Token	eatWhitespaceAndComment(StringBuffer sb)
Token	getToken()
void	next()
protected void	resolve()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

_tokenizer

protected final BufferedTokenizer _tokenizer

_token

protected Token _token

Constructor Detail

RegroupingTokenizer

public RegroupingTokenizer(Tokenizer tokenizer)

Method Detail

getToken

public Token getToken()

Specified by:

getToken in interface Tokenizer

next

public void next()

Specified by:

next in interface Tokenizer

eatWhitespaceAndComment

protected Token eatWhitespaceAndComment(StringBuffer sb)

resolve

protected void resolve()