org.apache.lucene.analysis
Class LowerCaseTokenizer
java.lang.Object
org.apache.lucene.util.AttributeSource
org.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.Tokenizer
org.apache.lucene.analysis.CharTokenizer
org.apache.lucene.analysis.LetterTokenizer
org.apache.lucene.analysis.LowerCaseTokenizer
public final class LowerCaseTokenizer
- extends LetterTokenizer
LowerCaseTokenizer performs the function of LetterTokenizer
and LowerCaseFilter together. It divides text at non-letters and converts
them to lower case. While it is functionally equivalent to the combination
of LetterTokenizer and LowerCaseFilter, there is a performance advantage
to doing the two tasks at once, hence this (redundant) implementation.
Note: this does a decent job for most European languages, but does a terrible
job for some Asian languages, where words are not separated by spaces.
Fields inherited from class org.apache.lucene.analysis.Tokenizer |
input |
Methods inherited from class org.apache.lucene.util.AttributeSource |
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, restoreState, toString |
LowerCaseTokenizer
public LowerCaseTokenizer(Reader in)
- Construct a new LowerCaseTokenizer.
LowerCaseTokenizer
public LowerCaseTokenizer(AttributeSource source,
Reader in)
- Construct a new LowerCaseTokenizer using a given
AttributeSource
.
LowerCaseTokenizer
public LowerCaseTokenizer(AttributeSource.AttributeFactory factory,
Reader in)
- Construct a new LowerCaseTokenizer using a given
AttributeSource.AttributeFactory
.
normalize
protected char normalize(char c)
- Converts char to lower case
Character.toLowerCase(char)
.
- Overrides:
normalize
in class CharTokenizer
Copyright © 2000-2010 Apache Software Foundation. All Rights Reserved.