|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object org.apache.lucene.util.AttributeSource org.apache.lucene.analysis.TokenStream
public abstract class TokenStream
A TokenStream
enumerates the sequence of tokens, either from
Field
s of a Document
or from query text.
This is an abstract class; concrete subclasses are:
Tokenizer
, a TokenStream
whose input is a Reader; and
TokenFilter
, a TokenStream
whose input is another
TokenStream
.
TokenStream
API has been introduced with Lucene 2.9. This API
has moved from being Token
-based to Attribute
-based. While
Token
still exists in 2.9 as a convenience class, the preferred way
to store the information of a Token
is to use AttributeImpl
s.
TokenStream
now extends AttributeSource
, which provides
access to all of the token Attribute
s for the TokenStream
.
Note that only one instance per AttributeImpl
is created and reused
for every token. This approach reduces object creation and allows local
caching of references to the AttributeImpl
s. See
incrementToken()
for further details.
The workflow of the new TokenStream
API is as follows:
TokenStream
/TokenFilter
s which add/get
attributes to/from the AttributeSource
.
reset()
.
incrementToken()
until it returns false
consuming the attributes after each call.
end()
so that any end-of-stream operations
can be performed.
close()
to release any resource when finished
using the TokenStream
.
incrementToken()
.
You can find some example code for the new API in the analysis package level Javadoc.
Sometimes it is desirable to capture a current state of a TokenStream
,
e.g., for buffering purposes (see CachingTokenFilter
,
TeeSinkTokenFilter
). For this usecase
AttributeSource.captureState()
and AttributeSource.restoreState(org.apache.lucene.util.AttributeSource.State)
can be used.
Nested Class Summary |
---|
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource |
---|
AttributeSource.AttributeFactory, AttributeSource.State |
Constructor Summary | |
---|---|
protected |
TokenStream()
A TokenStream using the default attribute factory. |
protected |
TokenStream(AttributeSource.AttributeFactory factory)
A TokenStream using the supplied AttributeFactory for creating new Attribute instances. |
protected |
TokenStream(AttributeSource input)
A TokenStream that uses the same attributes as the supplied one. |
Method Summary | |
---|---|
void |
close()
Releases resources associated with this stream. |
void |
end()
This method is called by the consumer after the last token has been consumed, after incrementToken() returned false
(using the new TokenStream API). |
static boolean |
getOnlyUseNewAPI()
Deprecated. This setting will no longer be needed in Lucene 3.0 as the old API will be removed. |
boolean |
incrementToken()
Consumers (i.e., IndexWriter ) use this method to advance the stream to
the next token. |
Token |
next()
Deprecated. The returned Token is a "full private copy" (not re-used across calls to next() ) but will be slower than calling
next(Token) or using the new incrementToken()
method with the new AttributeSource API. |
Token |
next(Token reusableToken)
Deprecated. The new incrementToken() and AttributeSource
APIs should be used instead. |
void |
reset()
Resets this stream to the beginning. |
static void |
setOnlyUseNewAPI(boolean onlyUseNewAPI)
Deprecated. This setting will no longer be needed in Lucene 3.0 as the old API will be removed. |
Methods inherited from class org.apache.lucene.util.AttributeSource |
---|
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, restoreState, toString |
Methods inherited from class java.lang.Object |
---|
clone, finalize, getClass, notify, notifyAll, wait, wait, wait |
Constructor Detail |
---|
protected TokenStream()
protected TokenStream(AttributeSource input)
protected TokenStream(AttributeSource.AttributeFactory factory)
Attribute
instances.
Method Detail |
---|
public static void setOnlyUseNewAPI(boolean onlyUseNewAPI)
incrementToken()
API using Attribute
s. There will be a
small, but in most cases negligible performance increase by enabling this,
but it only works if all TokenStream
s use the new API and
implement incrementToken()
. This setting can only be enabled
globally.
This setting only affects TokenStream
s instantiated after this
call. All TokenStream
s already created use the other setting.
All core Analyzer
s are compatible with this setting, if you have
your own TokenStream
s that are also compatible, you should enable
this.
When enabled, tokenization may throw UnsupportedOperationException
s, if the whole tokenizer chain is not compatible eg one of the
TokenStream
s does not implement the new TokenStream
API.
The default is false
, so there is the fallback to the old API
available.
public static boolean getOnlyUseNewAPI()
setOnlyUseNewAPI(boolean)
public boolean incrementToken() throws IOException
IndexWriter
) use this method to advance the stream to
the next token. Implementing classes must implement this method and update
the appropriate AttributeImpl
s with the attributes of the next
token.
The producer must make no assumptions about the attributes after the method
has been returned: the caller may arbitrarily change it. If the producer
needs to preserve the state for subsequent calls, it can use
AttributeSource.captureState()
to create a copy of the current attribute state.
This method is called for every token of a document, so an efficient
implementation is crucial for good performance. To avoid calls to
AttributeSource.addAttribute(Class)
and AttributeSource.getAttribute(Class)
or downcasts,
references to all AttributeImpl
s that this stream uses should be
retrieved during instantiation.
To ensure that filters and consumers know which attributes are available,
the attributes must be added during instantiation. Filters and consumers
are not required to check for availability of attributes in
incrementToken()
.
Note that this method will be defined abstract in Lucene 3.0.
IOException
public void end() throws IOException
incrementToken()
returned false
(using the new TokenStream
API). Streams implementing the old API
should upgrade to use this feature.
This method can be used to perform any end-of-stream operations, such as
setting the final offset of a stream. The final offset of a stream might
differ from the offset of the last token eg in case one or more whitespaces
followed after the last token, but a WhitespaceTokenizer
was used.
IOException
public Token next(Token reusableToken) throws IOException
incrementToken()
and AttributeSource
APIs should be used instead.
This implicitly defines a "contract" between consumers (callers of this method) and producers (implementations of this method that are the source for tokens):
Token
before calling this method again.Token.clear()
before setting the fields in
it and returning itToken
after it
has been returned: the caller may arbitrarily change it. If the producer
needs to hold onto the Token
for subsequent calls, it must clone()
it before storing it. Note that a TokenFilter
is considered a
consumer.
reusableToken
- a Token
that may or may not be used to return;
this parameter should never be null (the callee is not required to
check for null before using it, but it is a good idea to assert that
it is not null.)
Token
in the stream or null if end-of-stream was hit
IOException
public Token next() throws IOException
next()
) but will be slower than calling
next(Token)
or using the new incrementToken()
method with the new AttributeSource
API.
Token
in the stream, or null at EOS.
IOException
public void reset() throws IOException
reset()
is not needed for
the standard indexing process. However, if the tokens of a
TokenStream
are intended to be consumed more than once, it is
necessary to implement reset()
. Note that if your TokenStream
caches tokens and feeds them back again after a reset, it is imperative
that you clone the tokens when you store them away (on the first pass) as
well as when you return them (on future passes after reset()
).
IOException
public void close() throws IOException
IOException
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |