org.apache.lucene.analysis
Class TokenStream

java.lang.Object
  extended by org.apache.lucene.util.AttributeSource
      extended by org.apache.lucene.analysis.TokenStream
Direct Known Subclasses:
EmptyTokenStream, NumericTokenStream, PrefixAndSuffixAwareTokenFilter, PrefixAwareTokenFilter, ShingleMatrixFilter, SingleTokenTokenStream, TeeSinkTokenFilter.SinkTokenStream, TokenFilter, Tokenizer

public abstract class TokenStream
extends AttributeSource

A TokenStream enumerates the sequence of tokens, either from Fields of a Document or from query text.

This is an abstract class; concrete subclasses are:

A new TokenStream API has been introduced with Lucene 2.9. This API has moved from being Token-based to Attribute-based. While Token still exists in 2.9 as a convenience class, the preferred way to store the information of a Token is to use AttributeImpls.

TokenStream now extends AttributeSource, which provides access to all of the token Attributes for the TokenStream. Note that only one instance per AttributeImpl is created and reused for every token. This approach reduces object creation and allows local caching of references to the AttributeImpls. See incrementToken() for further details.

The workflow of the new TokenStream API is as follows:

  1. Instantiation of TokenStream/TokenFilters which add/get attributes to/from the AttributeSource.
  2. The consumer calls reset().
  3. The consumer retrieves attributes from the stream and stores local references to all attributes it wants to access.
  4. The consumer calls incrementToken() until it returns false consuming the attributes after each call.
  5. The consumer calls end() so that any end-of-stream operations can be performed.
  6. The consumer calls close() to release any resource when finished using the TokenStream.
To make sure that filters and consumers know which attributes are available, the attributes must be added during instantiation. Filters and consumers are not required to check for availability of attributes in incrementToken().

You can find some example code for the new API in the analysis package level Javadoc.

Sometimes it is desirable to capture a current state of a TokenStream, e.g., for buffering purposes (see CachingTokenFilter, TeeSinkTokenFilter). For this usecase AttributeSource.captureState() and AttributeSource.restoreState(org.apache.lucene.util.AttributeSource.State) can be used.


Nested Class Summary
 
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
AttributeSource.AttributeFactory, AttributeSource.State
 
Constructor Summary
protected TokenStream()
          A TokenStream using the default attribute factory.
protected TokenStream(AttributeSource.AttributeFactory factory)
          A TokenStream using the supplied AttributeFactory for creating new Attribute instances.
protected TokenStream(AttributeSource input)
          A TokenStream that uses the same attributes as the supplied one.
 
Method Summary
 void close()
          Releases resources associated with this stream.
 void end()
          This method is called by the consumer after the last token has been consumed, after incrementToken() returned false (using the new TokenStream API).
static boolean getOnlyUseNewAPI()
          Deprecated. This setting will no longer be needed in Lucene 3.0 as the old API will be removed.
 boolean incrementToken()
          Consumers (i.e., IndexWriter) use this method to advance the stream to the next token.
 Token next()
          Deprecated. The returned Token is a "full private copy" (not re-used across calls to next()) but will be slower than calling next(Token) or using the new incrementToken() method with the new AttributeSource API.
 Token next(Token reusableToken)
          Deprecated. The new incrementToken() and AttributeSource APIs should be used instead.
 void reset()
          Resets this stream to the beginning.
static void setOnlyUseNewAPI(boolean onlyUseNewAPI)
          Deprecated. This setting will no longer be needed in Lucene 3.0 as the old API will be removed.
 
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, restoreState, toString
 
Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait
 

Constructor Detail

TokenStream

protected TokenStream()
A TokenStream using the default attribute factory.


TokenStream

protected TokenStream(AttributeSource input)
A TokenStream that uses the same attributes as the supplied one.


TokenStream

protected TokenStream(AttributeSource.AttributeFactory factory)
A TokenStream using the supplied AttributeFactory for creating new Attribute instances.

Method Detail

setOnlyUseNewAPI

public static void setOnlyUseNewAPI(boolean onlyUseNewAPI)
Deprecated. This setting will no longer be needed in Lucene 3.0 as the old API will be removed.

For extra performance you can globally enable the new incrementToken() API using Attributes. There will be a small, but in most cases negligible performance increase by enabling this, but it only works if all TokenStreams use the new API and implement incrementToken(). This setting can only be enabled globally.

This setting only affects TokenStreams instantiated after this call. All TokenStreams already created use the other setting.

All core Analyzers are compatible with this setting, if you have your own TokenStreams that are also compatible, you should enable this.

When enabled, tokenization may throw UnsupportedOperationException s, if the whole tokenizer chain is not compatible eg one of the TokenStreams does not implement the new TokenStream API.

The default is false, so there is the fallback to the old API available.


getOnlyUseNewAPI

public static boolean getOnlyUseNewAPI()
Deprecated. This setting will no longer be needed in Lucene 3.0 as the old API will be removed.

Returns if only the new API is used.

See Also:
setOnlyUseNewAPI(boolean)

incrementToken

public boolean incrementToken()
                       throws IOException
Consumers (i.e., IndexWriter) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriate AttributeImpls with the attributes of the next token.

The producer must make no assumptions about the attributes after the method has been returned: the caller may arbitrarily change it. If the producer needs to preserve the state for subsequent calls, it can use AttributeSource.captureState() to create a copy of the current attribute state.

This method is called for every token of a document, so an efficient implementation is crucial for good performance. To avoid calls to AttributeSource.addAttribute(Class) and AttributeSource.getAttribute(Class) or downcasts, references to all AttributeImpls that this stream uses should be retrieved during instantiation.

To ensure that filters and consumers know which attributes are available, the attributes must be added during instantiation. Filters and consumers are not required to check for availability of attributes in incrementToken().

Returns:
false for end of stream; true otherwise

Note that this method will be defined abstract in Lucene 3.0.

Throws:
IOException

end

public void end()
         throws IOException
This method is called by the consumer after the last token has been consumed, after incrementToken() returned false (using the new TokenStream API). Streams implementing the old API should upgrade to use this feature.

This method can be used to perform any end-of-stream operations, such as setting the final offset of a stream. The final offset of a stream might differ from the offset of the last token eg in case one or more whitespaces followed after the last token, but a WhitespaceTokenizer was used.

Throws:
IOException

next

public Token next(Token reusableToken)
           throws IOException
Deprecated. The new incrementToken() and AttributeSource APIs should be used instead.

Returns the next token in the stream, or null at EOS. When possible, the input Token should be used as the returned Token (this gives fastest tokenization performance), but this is not required and a new Token may be returned. Callers may re-use a single Token instance for successive calls to this method.

This implicitly defines a "contract" between consumers (callers of this method) and producers (implementations of this method that are the source for tokens):

Also, the producer must make no assumptions about a Token after it has been returned: the caller may arbitrarily change it. If the producer needs to hold onto the Token for subsequent calls, it must clone() it before storing it. Note that a TokenFilter is considered a consumer.

Parameters:
reusableToken - a Token that may or may not be used to return; this parameter should never be null (the callee is not required to check for null before using it, but it is a good idea to assert that it is not null.)
Returns:
next Token in the stream or null if end-of-stream was hit
Throws:
IOException

next

public Token next()
           throws IOException
Deprecated. The returned Token is a "full private copy" (not re-used across calls to next()) but will be slower than calling next(Token) or using the new incrementToken() method with the new AttributeSource API.

Returns the next Token in the stream, or null at EOS.

Throws:
IOException

reset

public void reset()
           throws IOException
Resets this stream to the beginning. This is an optional operation, so subclasses may or may not implement this method. reset() is not needed for the standard indexing process. However, if the tokens of a TokenStream are intended to be consumed more than once, it is necessary to implement reset(). Note that if your TokenStream caches tokens and feeds them back again after a reset, it is imperative that you clone the tokens when you store them away (on the first pass) as well as when you return them (on future passes after reset()).

Throws:
IOException

close

public void close()
           throws IOException
Releases resources associated with this stream.

Throws:
IOException


Copyright © 2000-2010 Apache Software Foundation. All Rights Reserved.