org.apache.lucene.analysis.miscellaneous
Class PatternAnalyzer

java.lang.Object
  extended by org.apache.lucene.analysis.Analyzer
      extended by org.apache.lucene.analysis.miscellaneous.PatternAnalyzer
All Implemented Interfaces:
Closeable

Deprecated. (4.0) use the pattern-based analysis in the analysis/pattern package instead.

@Deprecated
public final class PatternAnalyzer
extends Analyzer

Efficient Lucene analyzer/tokenizer that preferably operates on a String rather than a Reader, that can flexibly separate text into terms via a regular expression Pattern (with behaviour identical to String.split(String)), and that combines the functionality of LetterTokenizer, LowerCaseTokenizer, WhitespaceTokenizer, StopFilter into a single efficient multi-purpose class.

If you are unsure how exactly a regular expression should look like, consider prototyping by simply trying various expressions on some test texts via String.split(String). Once you are satisfied, give that regex to PatternAnalyzer. Also see Java Regular Expression Tutorial.

This class can be considerably faster than the "normal" Lucene tokenizers. It can also serve as a building block in a compound Lucene TokenFilter chain. For example as in this stemming example:

 PatternAnalyzer pat = ...
 TokenStream tokenStream = new SnowballFilter(
     pat.tokenStream("content", "James is running round in the woods"), 
     "English"));
 


Nested Class Summary
 
Nested classes/interfaces inherited from class org.apache.lucene.analysis.Analyzer
Analyzer.GlobalReuseStrategy, Analyzer.PerFieldReuseStrategy, Analyzer.ReuseStrategy, Analyzer.TokenStreamComponents
 
Field Summary
static PatternAnalyzer DEFAULT_ANALYZER
          Deprecated. A lower-casing word analyzer with English stop words (can be shared freely across threads without harm); global per class loader.
static PatternAnalyzer EXTENDED_ANALYZER
          Deprecated. A lower-casing word analyzer with extended English stop words (can be shared freely across threads without harm); global per class loader.
static Pattern NON_WORD_PATTERN
          Deprecated. "\\W+"; Divides text at non-letters (NOT Character.isLetter(c))
static Pattern WHITESPACE_PATTERN
          Deprecated. "\\s+"; Divides text at whitespaces (Character.isWhitespace(c))
 
Constructor Summary
PatternAnalyzer(Version matchVersion, Pattern pattern, boolean toLowerCase, CharArraySet stopWords)
          Deprecated. Constructs a new instance with the given parameters.
 
Method Summary
 Analyzer.TokenStreamComponents createComponents(String fieldName, Reader reader)
          Deprecated. Creates a token stream that tokenizes all the text in the given Reader; This implementation forwards to tokenStream(String, Reader, String) and is less efficient than tokenStream(String, Reader, String).
 Analyzer.TokenStreamComponents createComponents(String fieldName, Reader reader, String text)
          Deprecated. Creates a token stream that tokenizes the given string into token terms (aka words).
 boolean equals(Object other)
          Deprecated. Indicates whether some other object is "equal to" this one.
 int hashCode()
          Deprecated. Returns a hash code value for the object.
 
Methods inherited from class org.apache.lucene.analysis.Analyzer
close, getOffsetGap, getPositionIncrementGap, initReader, tokenStream
 
Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

NON_WORD_PATTERN

public static final Pattern NON_WORD_PATTERN
Deprecated. 
"\\W+"; Divides text at non-letters (NOT Character.isLetter(c))


WHITESPACE_PATTERN

public static final Pattern WHITESPACE_PATTERN
Deprecated. 
"\\s+"; Divides text at whitespaces (Character.isWhitespace(c))


DEFAULT_ANALYZER

public static final PatternAnalyzer DEFAULT_ANALYZER
Deprecated. 
A lower-casing word analyzer with English stop words (can be shared freely across threads without harm); global per class loader.


EXTENDED_ANALYZER

public static final PatternAnalyzer EXTENDED_ANALYZER
Deprecated. 
A lower-casing word analyzer with extended English stop words (can be shared freely across threads without harm); global per class loader. The stop words are borrowed from http://thomas.loc.gov/home/stopwords.html, see http://thomas.loc.gov/home/all.about.inquery.html

Constructor Detail

PatternAnalyzer

public PatternAnalyzer(Version matchVersion,
                       Pattern pattern,
                       boolean toLowerCase,
                       CharArraySet stopWords)
Deprecated. 
Constructs a new instance with the given parameters.

Parameters:
matchVersion - currently does nothing
pattern - a regular expression delimiting tokens
toLowerCase - if true returns tokens after applying String.toLowerCase()
stopWords - if non-null, ignores all tokens that are contained in the given stop set (after previously having applied toLowerCase() if applicable). For example, created via StopFilter.makeStopSet(Version, String[])and/or WordlistLoaderas in WordlistLoader.getWordSet(new File("samples/fulltext/stopwords.txt") or other stop words lists .
Method Detail

createComponents

public Analyzer.TokenStreamComponents createComponents(String fieldName,
                                                       Reader reader,
                                                       String text)
Deprecated. 
Creates a token stream that tokenizes the given string into token terms (aka words).

Parameters:
fieldName - the name of the field to tokenize (currently ignored).
reader - reader (e.g. charfilter) of the original text. can be null.
text - the string to tokenize
Returns:
a new token stream

createComponents

public Analyzer.TokenStreamComponents createComponents(String fieldName,
                                                       Reader reader)
Deprecated. 
Creates a token stream that tokenizes all the text in the given Reader; This implementation forwards to tokenStream(String, Reader, String) and is less efficient than tokenStream(String, Reader, String).

Specified by:
createComponents in class Analyzer
Parameters:
fieldName - the name of the field to tokenize (currently ignored).
reader - the reader delivering the text
Returns:
a new token stream

equals

public boolean equals(Object other)
Deprecated. 
Indicates whether some other object is "equal to" this one.

Overrides:
equals in class Object
Parameters:
other - the reference object with which to compare.
Returns:
true if equal, false otherwise

hashCode

public int hashCode()
Deprecated. 
Returns a hash code value for the object.

Overrides:
hashCode in class Object
Returns:
the hash code.


Copyright © 2000-2013 Apache Software Foundation. All Rights Reserved.