org.apache.solr.analysis
Class BufferedTokenStream

java.lang.Object
  org.apache.lucene.util.AttributeSource
      org.apache.lucene.analysis.TokenStream
          org.apache.lucene.analysis.TokenFilter
              org.apache.solr.analysis.BufferedTokenStream

All Implemented Interfaces:

java.io.Closeable

Direct Known Subclasses:

CommonGramsQueryFilter, RemoveDuplicatesTokenFilter

Deprecated. This class does not support custom attributes. Extend TokenFilter instead, using AttributeSource.captureState() and AttributeSource#restoreState() which support all attributes.

@Deprecated
public abstract class BufferedTokenStream
extends org.apache.lucene.analysis.TokenFilter

Handles input and output buffering of TokenStream

 // Example of a class implementing the rule "A" "B" => "Q" "B"
 class MyTokenStream extends BufferedTokenStream {
   public MyTokenStream(TokenStream input) {super(input);}
   protected Token process(Token t) throws IOException {
     if ("A".equals(t.termText())) {
       Token t2 = read();
       if (t2!=null && "B".equals(t2.termText())) t.setTermText("Q");
       if (t2!=null) pushBack(t2);
          }
     return t;
   }
 }
 

 // Example of a class implementing "A" "B" => "A" "A" "B"
 class MyTokenStream extends BufferedTokenStream {
   public MyTokenStream(TokenStream input) {super(input);}
   protected Token process(Token t) throws IOException {
     if ("A".equals(t.termText()) && "B".equals(peek(1).termText()))
       write((Token)t.clone());
     return t;
   }
 }
 

NOTE: BufferedTokenStream does not clone() any Tokens. This is instead the responsibility of the implementing subclass. In the "A" "B" => "A" "A" "B" example above, the subclass must clone the additional "A" it creates.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

org.apache.lucene.util.AttributeSource.AttributeFactory, org.apache.lucene.util.AttributeSource.State

Field Summary

Fields inherited from class org.apache.lucene.analysis.TokenFilter

input

Constructor Summary

BufferedTokenStream(org.apache.lucene.analysis.TokenStream input)
Deprecated.

Method Summary

boolean	incrementToken() Deprecated.
protected java.lang.Iterable<org.apache.lucene.analysis.Token>	output() Deprecated. Provides direct Iterator access to the buffered output stream.
protected org.apache.lucene.analysis.Token	peek(int n) Deprecated. Peek n tokens ahead in the buffered input stream, without modifying the stream.
protected abstract org.apache.lucene.analysis.Token	process(org.apache.lucene.analysis.Token t) Deprecated. Process a token.
protected void	pushBack(org.apache.lucene.analysis.Token t) Deprecated. Push a token back into the buffered input stream, such that it will be returned by a future call to read()
protected org.apache.lucene.analysis.Token	read() Deprecated. Read a token from the buffered input stream.
void	reset() Deprecated.
protected void	write(org.apache.lucene.analysis.Token t) Deprecated. Write a token to the buffered output stream

Methods inherited from class org.apache.lucene.analysis.TokenFilter

close, end

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, restoreState, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

BufferedTokenStream

public BufferedTokenStream(org.apache.lucene.analysis.TokenStream input)

Deprecated.

Method Detail

process

protected abstract org.apache.lucene.analysis.Token process(org.apache.lucene.analysis.Token t)
                                                     throws java.io.IOException

Deprecated.

Process a token. Subclasses may read more tokens from the input stream, write more tokens to the output stream, or simply return the next token to be output. Subclasses may return null if the token is to be dropped. If a subclass writes tokens to the output stream and returns a non-null Token, the returned Token is considered to be at the head of the token output stream.

Throws:

java.io.IOException

incrementToken

public final boolean incrementToken()
                             throws java.io.IOException

Deprecated.

Specified by:

incrementToken in class org.apache.lucene.analysis.TokenStream

Throws:

java.io.IOException

read

protected org.apache.lucene.analysis.Token read()
                                         throws java.io.IOException

Deprecated.

Read a token from the buffered input stream.

Returns:

null at EOS

Throws:

java.io.IOException

pushBack

protected void pushBack(org.apache.lucene.analysis.Token t)

Deprecated.

Push a token back into the buffered input stream, such that it will be returned by a future call to read()

peek

protected org.apache.lucene.analysis.Token peek(int n)
                                         throws java.io.IOException

Deprecated.

Peek n tokens ahead in the buffered input stream, without modifying the stream.

Parameters:

n - Number of tokens into the input stream to peek, 1 based ... 0 is invalid

Returns:

a Token which exists in the input stream, any modifications made to this Token will be "real" if/when the Token is read() from the stream.

Throws:

java.io.IOException

write

protected void write(org.apache.lucene.analysis.Token t)

Deprecated.

Write a token to the buffered output stream

output

protected java.lang.Iterable<org.apache.lucene.analysis.Token> output()

Deprecated.

Provides direct Iterator access to the buffered output stream. Modifying any token in this Iterator will affect the resulting stream.

reset

public void reset()
           throws java.io.IOException

Deprecated.

Overrides:

reset in class org.apache.lucene.analysis.TokenFilter

Throws:

java.io.IOException