public final class LowerCaseTokenizer extends LetterTokenizer
Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces.
You must specify the required Version
compatibility when creating
LowerCaseTokenizer
:
CharTokenizer
uses an int based API to normalize and
detect token characters. See CharTokenizer.isTokenChar(int)
and
CharTokenizer.normalize(int)
for details.AttributeSource.AttributeFactory, AttributeSource.State
Constructor and Description |
---|
LowerCaseTokenizer(Version matchVersion,
AttributeSource.AttributeFactory factory,
Reader in)
Construct a new LowerCaseTokenizer using a given
AttributeSource.AttributeFactory . |
LowerCaseTokenizer(Version matchVersion,
Reader in)
Construct a new LowerCaseTokenizer.
|
Modifier and Type | Method and Description |
---|---|
protected int |
normalize(int c)
Converts char to lower case
Character.toLowerCase(int) . |
isTokenChar
end, incrementToken, reset
close, correctOffset, setReader
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState
public LowerCaseTokenizer(Version matchVersion, Reader in)
matchVersion
- Lucene version to match See abovein
- the input to split up into tokenspublic LowerCaseTokenizer(Version matchVersion, AttributeSource.AttributeFactory factory, Reader in)
AttributeSource.AttributeFactory
.protected int normalize(int c)
Character.toLowerCase(int)
.normalize
in class CharTokenizer
Copyright © 2000-2013 Apache Software Foundation. All Rights Reserved.