public final class WhitespaceTokenizer extends CharTokenizer
You must specify the required Version
compatibility when creating
WhitespaceTokenizer
:
CharTokenizer
uses an int based API to normalize and
detect token characters. See CharTokenizer.isTokenChar(int)
and
CharTokenizer.normalize(int)
for details.AttributeSource.AttributeFactory, AttributeSource.State
Constructor and Description |
---|
WhitespaceTokenizer(AttributeSource.AttributeFactory factory,
Reader in)
Deprecated.
use
WhitespaceTokenizer(Version, AttributeSource.AttributeFactory, Reader)
instead. This will be removed in Lucene 4.0. |
WhitespaceTokenizer(AttributeSource source,
Reader in)
Deprecated.
use
WhitespaceTokenizer(Version, AttributeSource, Reader)
instead. This will be removed in Lucene 4.0. |
WhitespaceTokenizer(Reader in)
Deprecated.
use
WhitespaceTokenizer(Version, Reader) instead. This will
be removed in Lucene 4.0. |
WhitespaceTokenizer(Version matchVersion,
AttributeSource.AttributeFactory factory,
Reader in)
Construct a new WhitespaceTokenizer using a given
AttributeSource.AttributeFactory . |
WhitespaceTokenizer(Version matchVersion,
AttributeSource source,
Reader in)
Construct a new WhitespaceTokenizer using a given
AttributeSource . |
WhitespaceTokenizer(Version matchVersion,
Reader in)
Construct a new WhitespaceTokenizer. * @param matchVersion Lucene version
to match See above
|
Modifier and Type | Method and Description |
---|---|
protected boolean |
isTokenChar(int c)
Collects only characters which do not satisfy
Character.isWhitespace(int) . |
end, incrementToken, isTokenChar, normalize, normalize, reset
close, correctOffset
reset
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString
public WhitespaceTokenizer(Version matchVersion, Reader in)
in
- the input to split up into tokenspublic WhitespaceTokenizer(Version matchVersion, AttributeSource source, Reader in)
AttributeSource
.public WhitespaceTokenizer(Version matchVersion, AttributeSource.AttributeFactory factory, Reader in)
AttributeSource.AttributeFactory
.@Deprecated public WhitespaceTokenizer(Reader in)
WhitespaceTokenizer(Version, Reader)
instead. This will
be removed in Lucene 4.0.@Deprecated public WhitespaceTokenizer(AttributeSource source, Reader in)
WhitespaceTokenizer(Version, AttributeSource, Reader)
instead. This will be removed in Lucene 4.0.AttributeSource
.@Deprecated public WhitespaceTokenizer(AttributeSource.AttributeFactory factory, Reader in)
WhitespaceTokenizer(Version, AttributeSource.AttributeFactory, Reader)
instead. This will be removed in Lucene 4.0.AttributeSource.AttributeFactory
.protected boolean isTokenChar(int c)
Character.isWhitespace(int)
.isTokenChar
in class CharTokenizer