StandardTokenizer
instead, which has the same functionality.
This filter will be removed in Lucene 5.0@Deprecated public final class ChineseTokenizer extends org.apache.lucene.analysis.Tokenizer
The difference between ChineseTokenizer and CJKTokenizer is that they have different token parsing logic.
For example, if the Chinese text "C1C2C3C4" is to be indexed:
Therefore the index created by CJKTokenizer is much larger.
The problem is that when searching for C1, C1C2, C1C3, C4C2, C1C2C3 ... the ChineseTokenizer works, but the CJKTokenizer will not work.
Constructor and Description |
---|
ChineseTokenizer(org.apache.lucene.util.AttributeSource.AttributeFactory factory,
Reader in)
Deprecated.
|
ChineseTokenizer(org.apache.lucene.util.AttributeSource source,
Reader in)
Deprecated.
|
ChineseTokenizer(Reader in)
Deprecated.
|
Modifier and Type | Method and Description |
---|---|
void |
end()
Deprecated.
|
boolean |
incrementToken()
Deprecated.
|
void |
reset()
Deprecated.
|
void |
reset(Reader input)
Deprecated.
|
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString
public ChineseTokenizer(Reader in)
public ChineseTokenizer(org.apache.lucene.util.AttributeSource source, Reader in)
public ChineseTokenizer(org.apache.lucene.util.AttributeSource.AttributeFactory factory, Reader in)
public boolean incrementToken() throws IOException
incrementToken
in class org.apache.lucene.analysis.TokenStream
IOException
public final void end()
end
in class org.apache.lucene.analysis.TokenStream
public void reset() throws IOException
reset
in class org.apache.lucene.analysis.TokenStream
IOException
public void reset(Reader input) throws IOException
reset
in class org.apache.lucene.analysis.Tokenizer
IOException