org.apache.lucene.analysis.ru
Class RussianLetterTokenizer
java.lang.Object
org.apache.lucene.util.AttributeSource
org.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.Tokenizer
org.apache.lucene.analysis.CharTokenizer
org.apache.lucene.analysis.ru.RussianLetterTokenizer
public class RussianLetterTokenizer
- extends CharTokenizer
A RussianLetterTokenizer is a Tokenizer
that extends LetterTokenizer
by additionally looking up letters in a given "russian charset".
The problem with
LetterTokenizer
is that it uses Character.isLetter(char)
method,
which doesn't know how to detect letters in encodings like CP1252 and KOI8
(well-known problems with 0xD7 and 0xF7 chars)
- Version:
- $Id: RussianLetterTokenizer.java 806961 2009-08-23 12:39:28Z rmuir $
Fields inherited from class org.apache.lucene.analysis.Tokenizer |
input |
Methods inherited from class org.apache.lucene.util.AttributeSource |
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, restoreState, toString |
RussianLetterTokenizer
public RussianLetterTokenizer(Reader in,
char[] charset)
- Deprecated. Use
RussianLetterTokenizer(Reader)
instead.
RussianLetterTokenizer
public RussianLetterTokenizer(Reader in)
RussianLetterTokenizer
public RussianLetterTokenizer(AttributeSource source,
Reader in)
RussianLetterTokenizer
public RussianLetterTokenizer(AttributeSource.AttributeFactory factory,
Reader in)
isTokenChar
protected boolean isTokenChar(char c)
- Collects only characters which satisfy
Character.isLetter(char)
.
- Specified by:
isTokenChar
in class CharTokenizer
Copyright © 2000-2010 Apache Software Foundation. All Rights Reserved.