org.apache.lucene.queries

Class CommonTermsQuery

java.lang.Object

org.apache.lucene.search.Query

org.apache.lucene.queries.CommonTermsQuery

public class CommonTermsQuery
extends Query

A query that executes high-frequency terms in a optional sub-query to prevent slow queries due to "common" terms like stopwords. This query builds 2 queries off the added terms: low-frequency terms are added to a required boolean clause and high-frequency terms are added to an optional boolean clause. The optional clause is only executed if the required "low-frequency" clause matches. In most cases, high-frequency terms are unlikely to significantly contribute to the document score unless at least one of the low-frequency terms are matched. This query can improve query execution times significantly if applicable.

CommonTermsQuery has several advantages over stopword filtering at index or query time since a term can be "classified" based on the actual document frequency in the index and can prevent slow queries even across domains without specialized stopword files.

Note: if the query only contains high-frequency terms the query is rewritten into a plain conjunction query ie. all high-frequency terms need to match in order to match a document.

Field Summary

Fields

Modifier and Type	Field and Description
protected float	highFreqBoost
protected float	highFreqMinNrShouldMatch
protected BooleanClause.Occur	highFreqOccur
protected float	lowFreqBoost
protected float	lowFreqMinNrShouldMatch
protected BooleanClause.Occur	lowFreqOccur
protected float	maxTermFrequency
protected List<Term>	terms

Constructor Summary

Constructors

Constructor and Description
CommonTermsQuery(BooleanClause.Occur highFreqOccur, BooleanClause.Occur lowFreqOccur, float maxTermFrequency) Creates a new CommonTermsQuery

Method Summary

Modifier and Type	Method and Description
void	add(Term term) Adds a term to the CommonTermsQuery
protected Query	buildQuery(int maxDoc, TermStates[] contextArray, Term[] queryTerms)
protected int	calcHighFreqMinimumNumberShouldMatch(int numOptional)
protected int	calcLowFreqMinimumNumberShouldMatch(int numOptional)
void	collectTermStates(IndexReader reader, List<LeafReaderContext> leaves, TermStates[] contextArray, Term[] queryTerms)
boolean	equals(Object other)
float	getHighFreqBoost() Gets the boost used for high frequency terms.
float	getHighFreqMinimumNumberShouldMatch() Gets the minimum number of the optional high frequent BooleanClauses which must be satisfied.
BooleanClause.Occur	getHighFreqOccur() Gets the BooleanClause.Occur used for high frequency terms.
float	getLowFreqBoost() Gets the boost used for low frequency terms.
float	getLowFreqMinimumNumberShouldMatch() Gets the minimum number of the optional low frequent BooleanClauses which must be satisfied.
BooleanClause.Occur	getLowFreqOccur() Gets the BooleanClause.Occur used for low frequency terms.
float	getMaxTermFrequency() Gets the maximum threshold of a terms document frequency to be considered a low frequency term.
List<Term>	getTerms() Gets the list of terms.
int	hashCode()
protected Query	newTermQuery(Term term, TermStates termStates) Builds a new TermQuery instance.
Query	rewrite(IndexReader reader)
void	setHighFreqMinimumNumberShouldMatch(float min) Specifies a minimum number of the high frequent optional BooleanClauses which must be satisfied in order to produce a match on the low frequency terms query part.
void	setLowFreqMinimumNumberShouldMatch(float min) Specifies a minimum number of the low frequent optional BooleanClauses which must be satisfied in order to produce a match on the low frequency terms query part.
String	toString(String field)

Methods inherited from class org.apache.lucene.search.Query

classHash, createWeight, sameClassAs, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

terms

protected final List<Term> terms

maxTermFrequency

protected final float maxTermFrequency

lowFreqOccur

protected final BooleanClause.Occur lowFreqOccur

highFreqOccur

protected final BooleanClause.Occur highFreqOccur

lowFreqBoost

protected float lowFreqBoost

highFreqBoost

protected float highFreqBoost

lowFreqMinNrShouldMatch

protected float lowFreqMinNrShouldMatch

highFreqMinNrShouldMatch

protected float highFreqMinNrShouldMatch

Constructor Detail

CommonTermsQuery

public CommonTermsQuery(BooleanClause.Occur highFreqOccur,
                        BooleanClause.Occur lowFreqOccur,
                        float maxTermFrequency)

Creates a new CommonTermsQuery

Parameters:

highFreqOccur - BooleanClause.Occur used for high frequency terms

lowFreqOccur - BooleanClause.Occur used for low frequency terms

maxTermFrequency - a value in [0..1) (or absolute number >=1) representing the maximum threshold of a terms document frequency to be considered a low frequency term.

Throws:

IllegalArgumentException - if BooleanClause.Occur.MUST_NOT is pass as lowFreqOccur or highFreqOccur

Method Detail

add

public void add(Term term)

Adds a term to the CommonTermsQuery

Parameters:

term - the term to add

rewrite

public Query rewrite(IndexReader reader)
              throws IOException

Overrides:

rewrite in class Query

Throws:

IOException

calcLowFreqMinimumNumberShouldMatch

protected int calcLowFreqMinimumNumberShouldMatch(int numOptional)

calcHighFreqMinimumNumberShouldMatch

protected int calcHighFreqMinimumNumberShouldMatch(int numOptional)

buildQuery

protected Query buildQuery(int maxDoc,
                           TermStates[] contextArray,
                           Term[] queryTerms)

collectTermStates

public void collectTermStates(IndexReader reader,
                              List<LeafReaderContext> leaves,
                              TermStates[] contextArray,
                              Term[] queryTerms)
                       throws IOException

Throws:

IOException

setLowFreqMinimumNumberShouldMatch

public void setLowFreqMinimumNumberShouldMatch(float min)

Specifies a minimum number of the low frequent optional BooleanClauses which must be satisfied in order to produce a match on the low frequency terms query part. This method accepts a float value in the range [0..1) as a fraction of the actual query terms in the low frequent clause or a number >=1 as an absolut number of clauses that need to match.

By default no optional clauses are necessary for a match (unless there are no required clauses). If this method is used, then the specified number of clauses is required.

Parameters:

min - the number of optional clauses that must match

getLowFreqMinimumNumberShouldMatch

public float getLowFreqMinimumNumberShouldMatch()

Gets the minimum number of the optional low frequent BooleanClauses which must be satisfied.

setHighFreqMinimumNumberShouldMatch

public void setHighFreqMinimumNumberShouldMatch(float min)

Specifies a minimum number of the high frequent optional BooleanClauses which must be satisfied in order to produce a match on the low frequency terms query part. This method accepts a float value in the range [0..1) as a fraction of the actual query terms in the low frequent clause or a number >=1 as an absolut number of clauses that need to match.

By default no optional clauses are necessary for a match (unless there are no required clauses). If this method is used, then the specified number of clauses is required.

Parameters:

min - the number of optional clauses that must match

getHighFreqMinimumNumberShouldMatch

public float getHighFreqMinimumNumberShouldMatch()

Gets the minimum number of the optional high frequent BooleanClauses which must be satisfied.

getTerms

public List<Term> getTerms()

Gets the list of terms.

getMaxTermFrequency

public float getMaxTermFrequency()

Gets the maximum threshold of a terms document frequency to be considered a low frequency term.

getLowFreqOccur

public BooleanClause.Occur getLowFreqOccur()

Gets the BooleanClause.Occur used for low frequency terms.

getHighFreqOccur

public BooleanClause.Occur getHighFreqOccur()

Gets the BooleanClause.Occur used for high frequency terms.

getLowFreqBoost

public float getLowFreqBoost()

Gets the boost used for low frequency terms.

getHighFreqBoost

public float getHighFreqBoost()

Gets the boost used for high frequency terms.

toString

public String toString(String field)

Specified by:

toString in class Query

hashCode

public int hashCode()

Specified by:

hashCode in class Query

equals

public boolean equals(Object other)

Specified by:

equals in class Query

newTermQuery

protected Query newTermQuery(Term term,
                             TermStates termStates)

Builds a new TermQuery instance.

This is intended for subclasses that wish to customize the generated queries.

Parameters:

term - term

termStates - the TermStates to be used to create the low level term query. Can be null.

Returns:

new TermQuery instance