|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object org.apache.lucene.util.AttributeSource org.apache.lucene.analysis.TokenStream org.apache.lucene.analysis.Tokenizer
public abstract class Tokenizer
A Tokenizer is a TokenStream whose input is a Reader.
This is an abstract class; subclasses must override TokenStream.incrementToken()
NOTE: Subclasses overriding TokenStream.incrementToken()
must
call AttributeSource.clearAttributes()
before
setting attributes.
Nested Class Summary |
---|
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource |
---|
AttributeSource.AttributeFactory, AttributeSource.State |
Field Summary | |
---|---|
protected Reader |
input
The text source for this Tokenizer. |
Constructor Summary | |
---|---|
protected |
Tokenizer(AttributeSource.AttributeFactory factory,
Reader input)
Construct a token stream processing the given input using the given AttributeFactory. |
protected |
Tokenizer(Reader input)
Construct a token stream processing the given input. |
Method Summary | |
---|---|
void |
close()
Releases resources associated with this stream. |
protected int |
correctOffset(int currentOff)
Return the corrected offset. |
void |
reset()
This method is called by a consumer before it begins consumption using TokenStream.incrementToken() . |
void |
setReader(Reader input)
Expert: Set a new reader on the Tokenizer. |
Methods inherited from class org.apache.lucene.analysis.TokenStream |
---|
end, incrementToken |
Methods inherited from class org.apache.lucene.util.AttributeSource |
---|
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString |
Methods inherited from class java.lang.Object |
---|
clone, finalize, getClass, notify, notifyAll, wait, wait, wait |
Field Detail |
---|
protected Reader input
Constructor Detail |
---|
protected Tokenizer(Reader input)
protected Tokenizer(AttributeSource.AttributeFactory factory, Reader input)
Method Detail |
---|
public void close() throws IOException
If you override this method, always call super.close()
, otherwise
some internal state will not be correctly reset (e.g., Tokenizer
will
throw IllegalStateException
on reuse).
NOTE:
The default implementation closes the input Reader, so
be sure to call super.close()
when overriding this method.
close
in interface Closeable
close
in class TokenStream
IOException
protected final int correctOffset(int currentOff)
input
is a CharFilter
subclass
this method calls CharFilter.correctOffset(int)
, else returns currentOff
.
currentOff
- offset as seen in the output
CharFilter.correctOffset(int)
public final void setReader(Reader input) throws IOException
IOException
public void reset() throws IOException
TokenStream
TokenStream.incrementToken()
.
Resets this stream to a clean state. Stateful implementations must implement this method so that they can be reused, just as if they had been created fresh.
If you override this method, always call super.reset()
, otherwise
some internal state will not be correctly reset (e.g., Tokenizer
will
throw IllegalStateException
on further usage).
reset
in class TokenStream
IOException
|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |