|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object org.apache.lucene.analysis.Analyzer org.apache.lucene.analysis.cn.smart.SmartChineseAnalyzer
public final class SmartChineseAnalyzer
SmartChineseAnalyzer is an analyzer for Chinese or mixed Chinese-English text. The analyzer uses probabilistic knowledge to find the optimal word segmentation for Simplified Chinese text. The text is first broken into sentences, then each sentence is segmented into words.
Segmentation is based upon the Hidden Markov Model. A large training corpus was used to calculate Chinese word frequency probability.
This analyzer requires a dictionary to provide statistical data. SmartChineseAnalyzer has an included dictionary out-of-box.
The included dictionary data is from ICTCLAS1.0. Thanks to ICTCLAS for their hard work, and for contributing the data under the Apache 2 License!
Nested Class Summary |
---|
Nested classes/interfaces inherited from class org.apache.lucene.analysis.Analyzer |
---|
Analyzer.GlobalReuseStrategy, Analyzer.PerFieldReuseStrategy, Analyzer.ReuseStrategy, Analyzer.TokenStreamComponents |
Field Summary |
---|
Fields inherited from class org.apache.lucene.analysis.Analyzer |
---|
GLOBAL_REUSE_STRATEGY, PER_FIELD_REUSE_STRATEGY |
Constructor Summary | |
---|---|
SmartChineseAnalyzer(Version matchVersion)
Create a new SmartChineseAnalyzer, using the default stopword list. |
|
SmartChineseAnalyzer(Version matchVersion,
boolean useDefaultStopWords)
Create a new SmartChineseAnalyzer, optionally using the default stopword list. |
|
SmartChineseAnalyzer(Version matchVersion,
CharArraySet stopWords)
Create a new SmartChineseAnalyzer, using the provided Set of stopwords. |
Method Summary | |
---|---|
Analyzer.TokenStreamComponents |
createComponents(String fieldName,
Reader reader)
|
static CharArraySet |
getDefaultStopSet()
Returns an unmodifiable instance of the default stop-words set. |
Methods inherited from class org.apache.lucene.analysis.Analyzer |
---|
close, getOffsetGap, getPositionIncrementGap, getReuseStrategy, initReader, tokenStream, tokenStream |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Constructor Detail |
---|
public SmartChineseAnalyzer(Version matchVersion)
public SmartChineseAnalyzer(Version matchVersion, boolean useDefaultStopWords)
Create a new SmartChineseAnalyzer, optionally using the default stopword list.
The included default stopword list is simply a list of punctuation. If you do not use this list, punctuation will not be removed from the text!
useDefaultStopWords
- true to use the default stopword list.public SmartChineseAnalyzer(Version matchVersion, CharArraySet stopWords)
Create a new SmartChineseAnalyzer, using the provided Set
of stopwords.
Note: the set should include punctuation, unless you want to index punctuation!
stopWords
- Set
of stopwords to use.Method Detail |
---|
public static CharArraySet getDefaultStopSet()
public Analyzer.TokenStreamComponents createComponents(String fieldName, Reader reader)
createComponents
in class Analyzer
|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |