Package org.apache.lucene.search.similarities

Class IBSimilarity

java.lang.Object

org.apache.lucene.search.similarities.Similarity

org.apache.lucene.search.similarities.SimilarityBase

org.apache.lucene.search.similarities.IBSimilarity

public class IBSimilarity
extends SimilarityBase

Provides a framework for the family of information-based models, as described in Stéphane Clinchant and Eric Gaussier. 2010. Information-based models for ad hoc IR. In Proceeding of the 33rd international ACM SIGIR conference on Research and development in information retrieval (SIGIR '10). ACM, New York, NY, USA, 234-241.

The retrieval function is of the form RSV(q, d) = ∑ -x^q_w log Prob(X_w ≥ t^d_w | λ_w), where

• x^q_w is the query boost;
• X_w is a random variable that counts the occurrences of word w;
• t^d_w is the normalized term frequency;
• λ_w is a parameter.

The framework described in the paper has many similarities to the DFR framework (see DFRSimilarity). It is possible that the two Similarities will be merged at one point.

To construct an IBSimilarity, you must specify the implementations for all three components of the Information-Based model.

1. Distribution: Probabilistic distribution used to model term occurrence
   • DistributionLL: Log-logistic
   • DistributionLL: Smoothed power-law
2. Lambda: λ_w parameter of the probability distribution
   • LambdaDF: Nw/N or average number of documents where w occurs
   • LambdaTTF: Fw/N or average number of occurrences of w in the collection
3. Normalization: Term frequency normalization

   Any supported DFR normalization (listed in DFRSimilarity)

See Also:

DFRSimilarity

WARNING: This API is experimental and might change in incompatible ways in the next release.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.search.similarities.Similarity

Similarity.SimScorer

Field Summary

Fields

Modifier and Type	Field	Description
protected Distribution	distribution	The probabilistic distribution used to model term occurrence.
protected Lambda	lambda	The lambda (λw) parameter.
protected Normalization	normalization	The term frequency normalization.

Constructor Summary

Constructors

Constructor	Description
IBSimilarity(Distribution distribution, Lambda lambda, Normalization normalization)	Creates IBSimilarity from the three components and using default discountOverlaps value.
IBSimilarity(Distribution distribution, Lambda lambda, Normalization normalization, boolean discountOverlaps)	Creates IBSimilarity from the three components and with the specified discountOverlaps value.

Method Summary

Modifier and Type	Method	Description
protected void	explain(List<Explanation> subs, BasicStats stats, double freq, double docLen)	Subclasses should implement this method to explain the score.
protected Explanation	explain(BasicStats stats, Explanation freq, double docLen)	Explains the score.
Distribution	getDistribution()	Returns the distribution
Lambda	getLambda()	Returns the distribution's lambda parameter
Normalization	getNormalization()	Returns the term frequency normalization
protected double	score(BasicStats stats, double freq, double docLen)	Scores the document doc.
String	toString()	The name of IB methods follow the pattern IB <distribution> <lambda><normalization>.

Methods inherited from class org.apache.lucene.search.similarities.SimilarityBase

fillBasicStats, log2, newStats, scorer

Methods inherited from class org.apache.lucene.search.similarities.Similarity

computeNorm, getDiscountOverlaps

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

distribution

protected final Distribution distribution

The probabilistic distribution used to model term occurrence.

lambda

protected final Lambda lambda

The lambda (λ_w) parameter.

normalization

protected final Normalization normalization

The term frequency normalization.

Constructor Detail

IBSimilarity

public IBSimilarity(Distribution distribution,
                    Lambda lambda,
                    Normalization normalization)

Creates IBSimilarity from the three components and using default discountOverlaps value.

Note that null values are not allowed: if you want no normalization, instead pass Normalization.NoNormalization.

Parameters:

distribution - probabilistic distribution modeling term occurrence

lambda - distribution's λ_w parameter

normalization - term frequency normalization

IBSimilarity

public IBSimilarity(Distribution distribution,
                    Lambda lambda,
                    Normalization normalization,
                    boolean discountOverlaps)

Creates IBSimilarity from the three components and with the specified discountOverlaps value.

Note that null values are not allowed: if you want no normalization, instead pass Normalization.NoNormalization.

Parameters:

distribution - probabilistic distribution modeling term occurrence

lambda - distribution's λ_w parameter

normalization - term frequency normalization

discountOverlaps - true if overlap tokens should not impact document length for scoring.

Method Detail

score

protected double score(BasicStats stats,
                       double freq,
                       double docLen)

Description copied from class: SimilarityBase

Scores the document doc.

Subclasses must apply their scoring formula in this class.

Specified by:

score in class SimilarityBase

Parameters:

stats - the corpus level statistics.

freq - the term frequency.

docLen - the document length.

Returns:

the score.

explain

protected void explain(List<Explanation> subs,
                       BasicStats stats,
                       double freq,
                       double docLen)

Description copied from class: SimilarityBase

Subclasses should implement this method to explain the score. expl already contains the score, the name of the class and the doc id, as well as the term frequency and its explanation; subclasses can add additional clauses to explain details of their scoring formulae.

The default implementation does nothing.

Overrides:

explain in class SimilarityBase

Parameters:

subs - the list of details of the explanation to extend

stats - the corpus level statistics.

freq - the term frequency.

docLen - the document length.

explain

protected Explanation explain(BasicStats stats,
                              Explanation freq,
                              double docLen)

Description copied from class: SimilarityBase

Explains the score. The implementation here provides a basic explanation in the format score(name-of-similarity, doc=doc-id, freq=term-frequency), computed from:, and attaches the score (computed via the SimilarityBase.score(BasicStats, double, double) method) and the explanation for the term frequency. Subclasses content with this format may add additional details in SimilarityBase.explain(List, BasicStats, double, double).

Overrides:

explain in class SimilarityBase

Parameters:

stats - the corpus level statistics.

freq - the term frequency and its explanation.

docLen - the document length.

Returns:

the explanation.

toString

public String toString()

The name of IB methods follow the pattern IB <distribution> <lambda><normalization>. The name of the distribution is the same as in the original paper; for the names of lambda parameters, refer to the javadoc of the Lambda classes.

Specified by:

toString in class SimilarityBase

getDistribution

public Distribution getDistribution()

Returns the distribution

getLambda

public Lambda getLambda()

Returns the distribution's lambda parameter

getNormalization

public Normalization getNormalization()

Returns the term frequency normalization