public class LetterTokenizer extends CharTokenizer
Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces.
You must specify the required Version
compatibility when creating
LetterTokenizer
:
CharTokenizer
uses an int based API to normalize and
detect token characters. See CharTokenizer.isTokenChar(int)
and
CharTokenizer.normalize(int)
for details.AttributeSource.State
DEFAULT_TOKEN_ATTRIBUTE_FACTORY
DEFAULT_ATTRIBUTE_FACTORY
Constructor and Description |
---|
LetterTokenizer(Version matchVersion,
AttributeFactory factory,
Reader in)
Construct a new LetterTokenizer using a given
AttributeFactory . |
LetterTokenizer(Version matchVersion,
Reader in)
Construct a new LetterTokenizer.
|
Modifier and Type | Method and Description |
---|---|
protected boolean |
isTokenChar(int c)
Collects only characters which satisfy
Character.isLetter(int) . |
end, incrementToken, normalize, reset
close, correctOffset, setReader
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString
public LetterTokenizer(Version matchVersion, Reader in)
matchVersion
- Lucene version to match See abovein
- the input to split up into tokenspublic LetterTokenizer(Version matchVersion, AttributeFactory factory, Reader in)
AttributeFactory
.protected boolean isTokenChar(int c)
Character.isLetter(int)
.isTokenChar
in class CharTokenizer
Copyright © 2000-2014 Apache Software Foundation. All Rights Reserved.