public final class LowerCaseTokenizer extends LetterTokenizer
Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces.
You must specify the required Version
compatibility when creating
LowerCaseTokenizer
:
CharTokenizer
uses an int based API to normalize and
detect token characters. See CharTokenizer.isTokenChar(int)
and
CharTokenizer.normalize(int)
for details.AttributeSource.State
DEFAULT_TOKEN_ATTRIBUTE_FACTORY
DEFAULT_ATTRIBUTE_FACTORY
Constructor and Description |
---|
LowerCaseTokenizer(Version matchVersion,
AttributeFactory factory,
Reader in)
Construct a new LowerCaseTokenizer using a given
AttributeFactory . |
LowerCaseTokenizer(Version matchVersion,
Reader in)
Construct a new LowerCaseTokenizer.
|
Modifier and Type | Method and Description |
---|---|
protected int |
normalize(int c)
Converts char to lower case
Character.toLowerCase(int) . |
isTokenChar
end, incrementToken, reset
close, correctOffset, setReader
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString
public LowerCaseTokenizer(Version matchVersion, Reader in)
matchVersion
- Lucene version to match See abovein
- the input to split up into tokenspublic LowerCaseTokenizer(Version matchVersion, AttributeFactory factory, Reader in)
AttributeFactory
.protected int normalize(int c)
Character.toLowerCase(int)
.normalize
in class CharTokenizer
Copyright © 2000-2014 Apache Software Foundation. All Rights Reserved.