org.apache.lucene.search.similarities

Class DFISimilarity

java.lang.Object

org.apache.lucene.search.similarities.Similarity

org.apache.lucene.search.similarities.SimilarityBase

org.apache.lucene.search.similarities.DFISimilarity

public class DFISimilarity
extends SimilarityBase

Implements the Divergence from Independence (DFI) model based on Chi-square statistics (i.e., standardized Chi-squared distance from independence in term frequency tf).

DFI is both parameter-free and non-parametric:

• parameter-free: it does not require any parameter tuning or training.
• non-parametric: it does not make any assumptions about word frequency distributions on document collections.

It is highly recommended not to remove stopwords (very common terms: the, of, and, to, a, in, for, is, on, that, etc) with this similarity.

For more information see: A nonparametric term weighting method for information retrieval based on measuring the divergence from independence

See Also:

IndependenceStandardized, IndependenceSaturated, IndependenceChiSquared

WARNING: This API is experimental and might change in incompatible ways in the next release.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.search.similarities.Similarity

Similarity.SimScorer

Field Summary

Fields inherited from class org.apache.lucene.search.similarities.SimilarityBase

discountOverlaps

Constructor Summary

Constructors

Constructor and Description
DFISimilarity(Independence independenceMeasure) Create DFI with the specified divergence from independence measure

Method Summary

Modifier and Type	Method and Description
protected Explanation	explain(BasicStats stats, Explanation freq, double docLen) Explains the score.
Independence	getIndependence() Returns the measure of independence
protected double	score(BasicStats stats, double freq, double docLen) Scores the document doc.
String	toString() Subclasses must override this method to return the name of the Similarity and preferably the values of parameters (if any) as well.

Methods inherited from class org.apache.lucene.search.similarities.SimilarityBase

computeNorm, explain, fillBasicStats, getDiscountOverlaps, log2, newStats, scorer, setDiscountOverlaps

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

DFISimilarity

public DFISimilarity(Independence independenceMeasure)

Create DFI with the specified divergence from independence measure

Parameters:

independenceMeasure - measure of divergence from independence

Method Detail

score

protected double score(BasicStats stats,
                       double freq,
                       double docLen)

Description copied from class: SimilarityBase

Scores the document doc.

Subclasses must apply their scoring formula in this class.

Specified by:

score in class SimilarityBase

Parameters:

stats - the corpus level statistics.

freq - the term frequency.

docLen - the document length.

Returns:

the score.

getIndependence

public Independence getIndependence()

Returns the measure of independence

explain

protected Explanation explain(BasicStats stats,
                              Explanation freq,
                              double docLen)

Description copied from class: SimilarityBase

Explains the score. The implementation here provides a basic explanation in the format score(name-of-similarity, doc=doc-id, freq=term-frequency), computed from:, and attaches the score (computed via the SimilarityBase.score(BasicStats, double, double) method) and the explanation for the term frequency. Subclasses content with this format may add additional details in SimilarityBase.explain(List, BasicStats, double, double).

Overrides:

explain in class SimilarityBase

Parameters:

stats - the corpus level statistics.

freq - the term frequency and its explanation.

docLen - the document length.

Returns:

the explanation.

toString

public String toString()

Description copied from class: SimilarityBase

Subclasses must override this method to return the name of the Similarity and preferably the values of parameters (if any) as well.

Specified by:

toString in class SimilarityBase