org.apache.lucene.analysis.cn.smart
Class SmartChineseAnalyzer

java.lang.Object
  extended by org.apache.lucene.analysis.Analyzer
      extended by org.apache.lucene.analysis.cn.smart.SmartChineseAnalyzer

public class SmartChineseAnalyzer
extends org.apache.lucene.analysis.Analyzer

SmartChineseAnalyzer is an analyzer for Chinese or mixed Chinese-English text. The analyzer uses probabilistic knowledge to find the optimal word segmentation for Simplified Chinese text. The text is first broken into sentences, then each sentence is segmented into words.

Segmentation is based upon the Hidden Markov Model. A large training corpus was used to calculate Chinese word frequency probability.

This analyzer requires a dictionary to provide statistical data. SmartChineseAnalyzer has an included dictionary out-of-box.

The included dictionary data is from ICTCLAS1.0. Thanks to ICTCLAS for their hard work, and for contributing the data under the Apache 2 License!

WARNING: The status of the analyzers/smartcn analysis.cn.smart package is experimental. The APIs and file formats introduced here might change in the future and will not be supported anymore in such a case.


Field Summary
 
Fields inherited from class org.apache.lucene.analysis.Analyzer
overridesTokenStreamMethod
 
Constructor Summary
SmartChineseAnalyzer()
          Deprecated. Use SmartChineseAnalyzer(Version) instead
SmartChineseAnalyzer(boolean useDefaultStopWords)
          Deprecated. Use SmartChineseAnalyzer(Version, boolean) instead
SmartChineseAnalyzer(Set stopWords)
          Deprecated. Use SmartChineseAnalyzer(Version, Set) instead
SmartChineseAnalyzer(org.apache.lucene.util.Version matchVersion)
          Create a new SmartChineseAnalyzer, using the default stopword list.
SmartChineseAnalyzer(org.apache.lucene.util.Version matchVersion, boolean useDefaultStopWords)
           Create a new SmartChineseAnalyzer, optionally using the default stopword list.
SmartChineseAnalyzer(org.apache.lucene.util.Version matchVersion, Set stopWords)
           Create a new SmartChineseAnalyzer, using the provided Set of stopwords.
 
Method Summary
 org.apache.lucene.analysis.TokenStream reusableTokenStream(String fieldName, Reader reader)
           
 org.apache.lucene.analysis.TokenStream tokenStream(String fieldName, Reader reader)
           
 
Methods inherited from class org.apache.lucene.analysis.Analyzer
close, getOffsetGap, getPositionIncrementGap, getPreviousTokenStream, setOverridesTokenStreamMethod, setPreviousTokenStream
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

SmartChineseAnalyzer

public SmartChineseAnalyzer()
Deprecated. Use SmartChineseAnalyzer(Version) instead

Create a new SmartChineseAnalyzer, using the default stopword list.


SmartChineseAnalyzer

public SmartChineseAnalyzer(org.apache.lucene.util.Version matchVersion)
Create a new SmartChineseAnalyzer, using the default stopword list.


SmartChineseAnalyzer

public SmartChineseAnalyzer(boolean useDefaultStopWords)
Deprecated. Use SmartChineseAnalyzer(Version, boolean) instead

Create a new SmartChineseAnalyzer, optionally using the default stopword list.

The included default stopword list is simply a list of punctuation. If you do not use this list, punctuation will not be removed from the text!

Parameters:
useDefaultStopWords - true to use the default stopword list.

SmartChineseAnalyzer

public SmartChineseAnalyzer(org.apache.lucene.util.Version matchVersion,
                            boolean useDefaultStopWords)

Create a new SmartChineseAnalyzer, optionally using the default stopword list.

The included default stopword list is simply a list of punctuation. If you do not use this list, punctuation will not be removed from the text!

Parameters:
useDefaultStopWords - true to use the default stopword list.

SmartChineseAnalyzer

public SmartChineseAnalyzer(Set stopWords)
Deprecated. Use SmartChineseAnalyzer(Version, Set) instead

Create a new SmartChineseAnalyzer, using the provided Set of stopwords.

Note: the set should include punctuation, unless you want to index punctuation!

Parameters:
stopWords - Set of stopwords to use.

SmartChineseAnalyzer

public SmartChineseAnalyzer(org.apache.lucene.util.Version matchVersion,
                            Set stopWords)

Create a new SmartChineseAnalyzer, using the provided Set of stopwords.

Note: the set should include punctuation, unless you want to index punctuation!

Parameters:
stopWords - Set of stopwords to use.
Method Detail

tokenStream

public org.apache.lucene.analysis.TokenStream tokenStream(String fieldName,
                                                          Reader reader)
Specified by:
tokenStream in class org.apache.lucene.analysis.Analyzer

reusableTokenStream

public org.apache.lucene.analysis.TokenStream reusableTokenStream(String fieldName,
                                                                  Reader reader)
                                                           throws IOException
Overrides:
reusableTokenStream in class org.apache.lucene.analysis.Analyzer
Throws:
IOException


Copyright © 2000-2010 Apache Software Foundation. All Rights Reserved.