org.apache.lucene.analysis.standard
Class StandardAnalyzer

java.lang.Object
  org.apache.lucene.analysis.Analyzer
      org.apache.lucene.analysis.standard.StandardAnalyzer

All Implemented Interfaces:

Closeable

public class StandardAnalyzer
extends Analyzer

Filters StandardTokenizer with StandardFilter, LowerCaseFilter and StopFilter, using a list of English stop words.

You must specify the required Version compatibility when creating StandardAnalyzer:

• As of 2.9, StopFilter preserves position increments
• As of 2.4, Tokens incorrectly identified as acronyms are corrected (see LUCENE-1608

Field Summary

static int	DEFAULT_MAX_TOKEN_LENGTH Default maximum allowed token length
static Set<?>	STOP_WORDS_SET An unmodifiable set containing some common English words that are usually not useful for searching.

Fields inherited from class org.apache.lucene.analysis.Analyzer

overridesTokenStreamMethod

Constructor Summary

StandardAnalyzer(Version matchVersion)
Builds an analyzer with the default stop words (STOP_WORDS_SET).

StandardAnalyzer(Version matchVersion, File stopwords)
Builds an analyzer with the stop words from the given file.

StandardAnalyzer(Version matchVersion, Reader stopwords)
Builds an analyzer with the stop words from the given reader.

StandardAnalyzer(Version matchVersion, Set<?> stopWords)
Builds an analyzer with the given stop words.

Method Summary

int	getMaxTokenLength()
TokenStream	reusableTokenStream(String fieldName, Reader reader) Creates a TokenStream that is allowed to be re-used from the previous time that the same thread called this method.
void	setMaxTokenLength(int length) Set maximum allowed token length.
TokenStream	tokenStream(String fieldName, Reader reader) Constructs a StandardTokenizer filtered by a StandardFilter, a LowerCaseFilter and a StopFilter.

Methods inherited from class org.apache.lucene.analysis.Analyzer

close, getOffsetGap, getPositionIncrementGap, getPreviousTokenStream, setOverridesTokenStreamMethod, setPreviousTokenStream

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

STOP_WORDS_SET

public static final Set<?> STOP_WORDS_SET

An unmodifiable set containing some common English words that are usually not useful for searching.

DEFAULT_MAX_TOKEN_LENGTH

public static final int DEFAULT_MAX_TOKEN_LENGTH

Default maximum allowed token length

See Also:

Constant Field Values

Constructor Detail

StandardAnalyzer

public StandardAnalyzer(Version matchVersion)

Builds an analyzer with the default stop words (STOP_WORDS_SET).

Parameters:

matchVersion - Lucene version to match See above

StandardAnalyzer

public StandardAnalyzer(Version matchVersion,
                        Set<?> stopWords)

Builds an analyzer with the given stop words.

Parameters:

matchVersion - Lucene version to match See above

stopWords - stop words

StandardAnalyzer

public StandardAnalyzer(Version matchVersion,
                        File stopwords)
                 throws IOException

Builds an analyzer with the stop words from the given file.

Parameters:

matchVersion - Lucene version to match See above

stopwords - File to read stop words from

Throws:

IOException

See Also:

WordlistLoader.getWordSet(File)

StandardAnalyzer

public StandardAnalyzer(Version matchVersion,
                        Reader stopwords)
                 throws IOException

Builds an analyzer with the stop words from the given reader.

Parameters:

matchVersion - Lucene version to match See above

stopwords - Reader to read stop words from

Throws:

IOException

See Also:

WordlistLoader.getWordSet(Reader)

Method Detail

tokenStream

public TokenStream tokenStream(String fieldName,
                               Reader reader)

Constructs a StandardTokenizer filtered by a StandardFilter, a LowerCaseFilter and a StopFilter.

Specified by:

tokenStream in class Analyzer

setMaxTokenLength

public void setMaxTokenLength(int length)

Set maximum allowed token length. If a token is seen that exceeds this length then it is discarded. This setting only takes effect the next time tokenStream or reusableTokenStream is called.

getMaxTokenLength

public int getMaxTokenLength()

See Also:

setMaxTokenLength(int)

reusableTokenStream

public TokenStream reusableTokenStream(String fieldName,
                                       Reader reader)
                                throws IOException

Description copied from class: Analyzer

Creates a TokenStream that is allowed to be re-used from the previous time that the same thread called this method. Callers that do not need to use more than one TokenStream at the same time from this analyzer should use this method for better performance.

Overrides:

reusableTokenStream in class Analyzer

Throws:

IOException