org.apache.solr.analysis
Class RemoveDuplicatesTokenFilter

java.lang.Object
  org.apache.lucene.util.AttributeSource
      org.apache.lucene.analysis.TokenStream
          org.apache.lucene.analysis.TokenFilter
              org.apache.solr.analysis.BufferedTokenStream
                  org.apache.solr.analysis.RemoveDuplicatesTokenFilter

All Implemented Interfaces:

java.io.Closeable

public class RemoveDuplicatesTokenFilter
extends BufferedTokenStream

A TokenFilter which filters out Tokens at the same position and Term text as the previous token in the stream.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

org.apache.lucene.util.AttributeSource.AttributeFactory, org.apache.lucene.util.AttributeSource.State

Field Summary

Fields inherited from class org.apache.lucene.analysis.TokenFilter

input

Constructor Summary

RemoveDuplicatesTokenFilter(org.apache.lucene.analysis.TokenStream input)

Method Summary

protected org.apache.lucene.analysis.Token

process(org.apache.lucene.analysis.Token t) Process a token.

Methods inherited from class org.apache.solr.analysis.BufferedTokenStream

incrementToken, output, peek, pushBack, read, reset, write

Methods inherited from class org.apache.lucene.analysis.TokenFilter

close, end

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, restoreState, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

RemoveDuplicatesTokenFilter

public RemoveDuplicatesTokenFilter(org.apache.lucene.analysis.TokenStream input)

Method Detail

process

protected org.apache.lucene.analysis.Token process(org.apache.lucene.analysis.Token t)
                                            throws java.io.IOException

Description copied from class: BufferedTokenStream

Process a token. Subclasses may read more tokens from the input stream, write more tokens to the output stream, or simply return the next token to be output. Subclasses may return null if the token is to be dropped. If a subclass writes tokens to the output stream and returns a non-null Token, the returned Token is considered to be at the head of the token output stream.

Specified by:

process in class BufferedTokenStream

Throws:

java.io.IOException