org.apache.lucene.search.similarities

Class DFISimilarity

java.lang.Object

org.apache.lucene.search.similarities.Similarity

org.apache.lucene.search.similarities.SimilarityBase

org.apache.lucene.search.similarities.DFISimilarity

public class DFISimilarity
extends SimilarityBase

Implements the Divergence from Independence (DFI) model based on Chi-square statistics (i.e., standardized Chi-squared distance from independence in term frequency tf).

DFI is both parameter-free and non-parametric:

• parameter-free: it does not require any parameter tuning or training.
• non-parametric: it does not make any assumptions about word frequency distributions on document collections.

It is highly recommended not to remove stopwords (very common terms: the, of, and, to, a, in, for, is, on, that, etc) with this similarity.

For more information see: A nonparametric term weighting method for information retrieval based on measuring the divergence from independence

See Also:

IndependenceStandardized, IndependenceSaturated, IndependenceChiSquared

WARNING: This API is experimental and might change in incompatible ways in the next release.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.search.similarities.Similarity

Similarity.SimScorer, Similarity.SimWeight

Field Summary

Fields inherited from class org.apache.lucene.search.similarities.SimilarityBase

discountOverlaps

Constructor Summary

Constructors

Constructor and Description
DFISimilarity(Independence independenceMeasure) Create DFI with the specified divergence from independence measure

Method Summary

Modifier and Type	Method and Description
Independence	getIndependence() Returns the measure of independence
protected float	score(BasicStats stats, float freq, float docLen) Scores the document doc.
String	toString() Subclasses must override this method to return the name of the Similarity and preferably the values of parameters (if any) as well.

Methods inherited from class org.apache.lucene.search.similarities.SimilarityBase

computeNorm, computeWeight, decodeNormValue, encodeNormValue, explain, explain, fillBasicStats, getDiscountOverlaps, log2, newStats, setDiscountOverlaps, simScorer

Methods inherited from class org.apache.lucene.search.similarities.Similarity

coord, queryNorm

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

DFISimilarity

public DFISimilarity(Independence independenceMeasure)

Create DFI with the specified divergence from independence measure

Parameters:

independenceMeasure - measure of divergence from independence

Method Detail

score

protected float score(BasicStats stats,
                      float freq,
                      float docLen)

Description copied from class: SimilarityBase

Scores the document doc.

Subclasses must apply their scoring formula in this class.

Specified by:

score in class SimilarityBase

Parameters:

stats - the corpus level statistics.

freq - the term frequency.

docLen - the document length.

Returns:

the score.

getIndependence

public Independence getIndependence()

Returns the measure of independence

toString

public String toString()

Description copied from class: SimilarityBase

Subclasses must override this method to return the name of the Similarity and preferably the values of parameters (if any) as well.

Specified by:

toString in class SimilarityBase