public abstract class Similarity extends Object
Expert: Scoring API.
This is a low-level API, you should only extend this API if you want to implement
an information retrieval model. If you are instead looking for a convenient way
to alter Lucene's scoring, consider extending a higher-level implementation
such as TFIDFSimilarity
, which implements the vector space model with this API, or
just tweaking the default implementation: BM25Similarity
.
Similarity determines how Lucene weights terms, and Lucene interacts with this class at both index-time and query-time.
Indexing Time
At indexing time, the indexer calls computeNorm(FieldInvertState)
, allowing
the Similarity implementation to set a per-document value for the field that will
be later accessible via LeafReader.getNormValues(String)
. Lucene makes no assumption
about what is in this norm, but it is most useful for encoding length normalization
information.
Implementations should carefully consider how the normalization is encoded: while
Lucene's BM25Similarity
encodes a combination of index-time boost
and length normalization information with SmallFloat
into a single byte, this
might not be suitable for all purposes.
Many formulas require the use of average document length, which can be computed via a
combination of CollectionStatistics.sumTotalTermFreq()
and
CollectionStatistics.maxDoc()
or CollectionStatistics.docCount()
,
depending upon whether the average should reflect field sparsity.
Additional scoring factors can be stored in named
NumericDocValuesField
s and accessed
at query-time with LeafReader.getNumericDocValues(String)
.
Finally, using index-time boosts (either via folding into the normalization byte or
via DocValues), is an inefficient way to boost the scores of different fields if the
boost will be the same for every document, instead the Similarity can simply take a constant
boost parameter C, and PerFieldSimilarityWrapper
can return different
instances with different boosts depending upon field name.
Query time At query-time, Queries interact with the Similarity via these steps:
computeWeight(CollectionStatistics, TermStatistics...)
method is called a single time,
allowing the implementation to compute any statistics (such as IDF, average document length, etc)
across the entire collection. The TermStatistics
and CollectionStatistics
passed in
already contain all of the raw statistics involved, so a Similarity can freely use any combination
of statistics without causing any additional I/O. Lucene makes no assumption about what is
stored in the returned Similarity.SimWeight
object.
Similarity.SimWeight.getValueForNormalization()
is called for each query leaf node, queryNorm(float)
is called for the top-level
query, and finally Similarity.SimWeight.normalize(float, float)
passes down the normalization value
and any top-level boosts (e.g. from enclosing BooleanQuery
s).
simScorer(SimWeight, org.apache.lucene.index.LeafReaderContext)
The score() method is called for each matching document.
Explanations
When IndexSearcher.explain(org.apache.lucene.search.Query, int)
is called, queries consult the Similarity's DocScorer for an
explanation of how it computed its score. The query passes in a the document id and an explanation of how the frequency
was computed.
IndexWriterConfig.setSimilarity(Similarity)
,
IndexSearcher.setSimilarity(Similarity)
Modifier and Type | Class and Description |
---|---|
static class |
Similarity.SimScorer
|
static class |
Similarity.SimWeight
Stores the weight for a query across the indexed collection.
|
Constructor and Description |
---|
Similarity()
Sole constructor.
|
Modifier and Type | Method and Description |
---|---|
abstract long |
computeNorm(FieldInvertState state)
Computes the normalization value for a field, given the accumulated
state of term processing for this field (see
FieldInvertState ). |
abstract Similarity.SimWeight |
computeWeight(CollectionStatistics collectionStats,
TermStatistics... termStats)
Compute any collection-level weight (e.g.
|
float |
coord(int overlap,
int maxOverlap)
Hook to integrate coordinate-level matching.
|
float |
queryNorm(float valueForNormalization)
Computes the normalization value for a query given the sum of the
normalized weights
Similarity.SimWeight.getValueForNormalization() of
each of the query terms. |
abstract Similarity.SimScorer |
simScorer(Similarity.SimWeight weight,
LeafReaderContext context)
Creates a new
Similarity.SimScorer to score matching documents from a segment of the inverted index. |
public Similarity()
public float coord(int overlap, int maxOverlap)
By default this is disabled (returns 1
), as with
most modern models this will only skew performance, but some
implementations such as TFIDFSimilarity
override this.
overlap
- the number of query terms matched in the documentmaxOverlap
- the total number of terms in the querypublic float queryNorm(float valueForNormalization)
Similarity.SimWeight.getValueForNormalization()
of
each of the query terms. This value is passed back to the
weight (Similarity.SimWeight.normalize(float, float)
of each query
term, to provide a hook to attempt to make scores from different
queries comparable.
By default this is disabled (returns 1
), but some
implementations such as TFIDFSimilarity
override this.
valueForNormalization
- the sum of the term normalization valuespublic abstract long computeNorm(FieldInvertState state)
FieldInvertState
).
Matches in longer fields are less precise, so implementations of this
method usually set smaller values when state.getLength()
is large,
and larger values when state.getLength()
is small.
state
- current processing state for this fieldpublic abstract Similarity.SimWeight computeWeight(CollectionStatistics collectionStats, TermStatistics... termStats)
collectionStats
- collection-level statistics, such as the number of tokens in the collection.termStats
- term-level statistics, such as the document frequency of a term across the collection.public abstract Similarity.SimScorer simScorer(Similarity.SimWeight weight, LeafReaderContext context) throws IOException
Similarity.SimScorer
to score matching documents from a segment of the inverted index.weight
- collection information from computeWeight(CollectionStatistics, TermStatistics...)
context
- segment of the inverted index to be scored.context
IOException
- if there is a low-level I/O errorCopyright © 2000-2017 Apache Software Foundation. All Rights Reserved.