org.apache.lucene.analysis.core
Class LetterTokenizer
java.lang.Object
org.apache.lucene.util.AttributeSource
org.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.Tokenizer
org.apache.lucene.analysis.util.CharTokenizer
org.apache.lucene.analysis.core.LetterTokenizer
- All Implemented Interfaces:
- Closeable
- Direct Known Subclasses:
- ArabicLetterTokenizer, LowerCaseTokenizer
public class LetterTokenizer
- extends CharTokenizer
A LetterTokenizer is a tokenizer that divides text at non-letters. That's to
say, it defines tokens as maximal strings of adjacent letters, as defined by
java.lang.Character.isLetter() predicate.
Note: this does a decent job for most European languages, but does a terrible
job for some Asian languages, where words are not separated by spaces.
You must specify the required Version
compatibility when creating
LetterTokenizer
:
Fields inherited from class org.apache.lucene.analysis.Tokenizer |
input |
Methods inherited from class org.apache.lucene.util.AttributeSource |
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState |
LetterTokenizer
public LetterTokenizer(Version matchVersion,
Reader in)
- Construct a new LetterTokenizer.
- Parameters:
matchVersion
- Lucene version to match See abovein
- the input to split up into tokens
LetterTokenizer
public LetterTokenizer(Version matchVersion,
AttributeSource source,
Reader in)
- Construct a new LetterTokenizer using a given
AttributeSource
.
- Parameters:
matchVersion
- Lucene version to match See abovesource
- the attribute source to use for this Tokenizer
in
- the input to split up into tokens
LetterTokenizer
public LetterTokenizer(Version matchVersion,
AttributeSource.AttributeFactory factory,
Reader in)
- Construct a new LetterTokenizer using a given
AttributeSource.AttributeFactory
.
- Parameters:
matchVersion
- Lucene version to match See abovefactory
- the attribute factory to use for this Tokenizer
in
- the input to split up into tokens
isTokenChar
protected boolean isTokenChar(int c)
- Collects only characters which satisfy
Character.isLetter(int)
.
- Specified by:
isTokenChar
in class CharTokenizer
Copyright © 2000-2013 Apache Software Foundation. All Rights Reserved.