public abstract class Tokenizer extends TokenStream
This is an abstract class; subclasses must override TokenStream.incrementToken()
NOTE: Subclasses overriding TokenStream.incrementToken()
must
call AttributeSource.clearAttributes()
before
setting attributes.
AttributeSource.State
Modifier and Type | Field and Description |
---|---|
protected Reader |
input
The text source for this Tokenizer.
|
DEFAULT_TOKEN_ATTRIBUTE_FACTORY
Modifier | Constructor and Description |
---|---|
protected |
Tokenizer()
Construct a tokenizer with no input, awaiting a call to
setReader(java.io.Reader)
to provide input. |
protected |
Tokenizer(AttributeFactory factory)
Construct a tokenizer with no input, awaiting a call to
setReader(java.io.Reader) to
provide input. |
Modifier and Type | Method and Description |
---|---|
void |
close()
Releases resources associated with this stream.
|
protected int |
correctOffset(int currentOff)
Return the corrected offset.
|
void |
reset()
This method is called by a consumer before it begins consumption using
TokenStream.incrementToken() . |
void |
setReader(Reader input)
Expert: Set a new reader on the Tokenizer.
|
end, incrementToken
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString
protected Reader input
protected Tokenizer()
setReader(java.io.Reader)
to provide input.protected Tokenizer(AttributeFactory factory)
setReader(java.io.Reader)
to
provide input.factory
- attribute factory.public void close() throws IOException
If you override this method, always call super.close()
, otherwise
some internal state will not be correctly reset (e.g., Tokenizer
will
throw IllegalStateException
on reuse).
NOTE:
The default implementation closes the input Reader, so
be sure to call super.close()
when overriding this method.
close
in interface Closeable
close
in interface AutoCloseable
close
in class TokenStream
IOException
protected final int correctOffset(int currentOff)
input
is a CharFilter
subclass
this method calls CharFilter.correctOffset(int)
, else returns currentOff
.currentOff
- offset as seen in the outputCharFilter.correctOffset(int)
public final void setReader(Reader input)
public void reset() throws IOException
TokenStream
TokenStream.incrementToken()
.
Resets this stream to a clean state. Stateful implementations must implement this method so that they can be reused, just as if they had been created fresh.
If you override this method, always call super.reset()
, otherwise
some internal state will not be correctly reset (e.g., Tokenizer
will
throw IllegalStateException
on further usage).
reset
in class TokenStream
IOException
Copyright © 2000-2016 Apache Software Foundation. All Rights Reserved.