Table Of Contents
- Search Basics
- The Query Classes
- Scoring: Introduction
- Scoring: Basics
- Changing the Scoring
- Appendix: Search Algorithm
Lucene offers a wide variety of
Query implementations, most of which are in
this package, its subpackage (
or the queries module. These implementations can be combined in a wide
variety of ways to provide complex querying capabilities along with information about where matches took place in the document
collection. The Query Classes section below highlights some of the more important Query classes. For details
on implementing your own Query class, see Custom Queries -- Expert Level below.
To perform a search, applications usually call
Once a Query has been created and submitted to the
IndexSearcher, the scoring
process begins. After some infrastructure setup, control finally passes to the
implementation and its
instances. See the Algorithm section for more notes on the process.
Of the various implementations of
is the easiest to understand and the most often used in applications. A
TermQuery matches all the documents that contain the
which is a word that occurs in a certain
TermQuery identifies and scores all
Documents that have a
Field with the specified string in it.
is as simple as:
TermQuery tq = new TermQuery(new Term("fieldName", "term"));In this example, the
Documents that have the
Fieldnamed "fieldName" containing the word "term".
Things start to get interesting when one combines multiple
TermQuery instances into a
BooleanQuery contains multiple
where each clause contains a sub-query (
instance) and an operator (from
describing how that sub-query is combined with the other clauses:
SHOULD— Use this operator when a clause can occur in the result set, but is not required. If a query is made up of all SHOULD clauses, then every document in the result set matches at least one of these clauses.
MUST— Use this operator when a clause is required to occur in the result set and should contribute to the score. Every document in the result set will match all such clauses.
FILTER— Use this operator when a clause is required to occur in the result set but should not contribute to the score. Every document in the result set will match all such clauses.
MUST NOT— Use this operator when a clause must not occur in the result set. No document in the result set will match any such clauses.
BooleanClauseinstances. If too many clauses are added, a
TooManyClausesexception will be thrown during searching. This most often occurs when a
Queryis rewritten into a
TermQueryclauses, for example by
WildcardQuery. The default setting for the maximum number of clauses is 1024, but this can be changed via the static method
Another common search is to find documents containing certain phrases. This is handled three different ways:
PhraseQuery— Matches a sequence of
PhraseQueryuses a slop factor to determine how many positions may occur between any two terms in the phrase and still be considered a match. The slop is 0 by default, meaning the phrase must match exactly.
MultiPhraseQuery— A more general form of PhraseQuery that accepts multiple Terms for a position in the phrase. For example, this can be used to perform phrase queries that also incorporate synonyms.
SpanNearQuery— Matches a sequence of other
SpanNearQueryallows for much more complicated phrase queries since it is constructed from other
SpanQueryinstances, instead of only
matches all documents that occur in a numeric range.
For PointRangeQuery to work, you must index the values
using a one of the numeric fields (
has a different implementation, it is essentially a special case of the
PrefixQuery allows an application
to identify all documents with terms that begin with a certain string. The
WildcardQuery generalizes this by allowing
for the use of * (matches 0 or more characters) and ? (matches exactly one character) wildcards.
Note that the
WildcardQuery can be quite slow. Also
not start with * and ?, as these are extremely slow.
Some QueryParsers may not allow this by default, but provide a
to remove that protection.
RegexpQuery is even more general than WildcardQuery,
allowing an application to identify all documents with terms that match a regular expression pattern.
matches documents that contain terms similar to the specified term. Similarity is
This type of query can be useful when accounting for spelling variations in the collection.
Scoring — Introduction
Lucene scoring is the heart of why we all love Lucene. It is blazingly fast and it hides almost all of the complexity from the user. In a nutshell, it works. At least, that is, until it doesn't work, or doesn't work as one would expect it to work. Then we are left digging into Lucene internals or asking for help on email@example.com to figure out why a document with five of our query terms scores lower than a different document with only one of the query terms.
While this document won't answer your specific scoring issues, it will, hopefully, point you to the places that can help you figure out the what and why of Lucene scoring.
Lucene scoring supports a number of pluggable information retrieval models, including:
Similarity API, and offer extension hooks and parameters for tuning. In general, Lucene first finds the documents that need to be scored based on boolean logic in the Query specification, and then ranks this subset of matching documents via the retrieval model. For some valuable references on VSM and IR in general refer to Lucene Wiki IR references.
The rest of this document will cover Scoring basics and explain how to
Similarity. Next, it will cover
ways you can customize the lucene internals in
Custom Queries -- Expert Level, which gives details on
implementing your own
Query class and related functionality.
Finally, we will finish up with some reference material in the Appendix.
Scoring — Basics
Scoring is very much dependent on the way documents are indexed, so it is important to understand
indexing. (see Lucene overview
before continuing on with this section) Be sure to use the useful
to understand how the score for a certain matching document was
Generally, the Query determines which documents match (a binary decision), while the Similarity determines how to assign scores to the matching documents.
Fields and Documents
In Lucene, the objects we are scoring are
A Document is a collection of
Fields. Each Field has
semantics about how it is created and stored
stored, etc). It is important to note that
Lucene scoring works on Fields and then combines the results to return Documents. This is
important because two Documents with the exact same content, but one having the content in two
Fields and the other in one Field may return different scores for the same query due to length
Lucene allows influencing the score contribution of various parts of the query by wrapping with
Changing Scoring — Similarity
Changing the scoring formula
Similarity is an easy way to
influence scoring, this is done at index-time with
IndexWriterConfig.setSimilarity(Similarity) and at query-time with
IndexSearcher.setSimilarity(Similarity). Be sure to use the same
Similarity at query-time as at index-time (so that norms are
encoded/decoded correctly); Lucene makes no effort to verify this.
You can influence scoring by configuring a different built-in Similarity implementation, or by tweaking its parameters, subclassing it to override behavior. Some implementations also offer a modular API which you can extend by plugging in a different component (e.g. term frequency normalizer).
Finally, you can extend the low level
to implement a new retrieval model.
org.apache.lucene.search.similarities package documentation for information
on the built-in available scoring models and extending or changing Similarity.
Integrating field values into the score
While similarities help score a document relatively to a query, it is also common for documents to hold
features that measure the quality of a match. Such features are best integrated into the score by indexing
FeatureField with the document at index-time, and then
combining the similarity score and the feature score using a linear combination. For instance the below
query matches the same documents as
originalQuery and computes scores as
similarityScore + 0.7 * featureScore:
Query originalQuery = new BooleanQuery.Builder() .add(new TermQuery(new Term("body", "apache")), Occur.SHOULD) .add(new TermQuery(new Term("body", "lucene")), Occur.SHOULD) .build(); Query featureQuery = FeatureField.newSaturationQuery("features", "pagerank"); Query query = new BooleanQuery.Builder() .add(originalQuery, Occur.MUST) .add(new BoostQuery(featureQuery, 0.7f), Occur.SHOULD) .build();
A less efficient yet more flexible way of modifying scores is to index scoring features into
doc-value fields and then combine them with the similarity score using a
from the queries module. For instance
the below example shows how to compute scores as
similarityScore * Math.log(popularity)
using the expressions module and
assuming that values for the
popularity field have been set in a
NumericDocValuesField at index time:
Custom Queries — Expert Level
Custom queries are an expert level task, so tread carefully and be prepared to share your code if you want help.
With the warning out of the way, it is possible to change a lot more than just the Similarity when it comes to matching and scoring in Lucene. Lucene's search is a complex mechanism that is grounded by three main classes:
Query— The abstract object representation of the user's information need.
Weight— A specialization of a Query for a given index. This typically associates a Query object with index statistics that are later used to compute document scores.
Scorer— The core class of the scoring process: for a given segment, scorers return
iteratorsover matches and give a way to compute the
scoreof these matches.
BulkScorer— An abstract class that scores a range of documents. A default implementation simply iterates through the hits from
Scorer, but some queries such as
BooleanQueryhave more efficient implementations.
The Query Class
In some sense, the
class is where it all begins. Without a Query, there would be
nothing to score. Furthermore, the Query class is the catalyst for the other scoring classes as it
is often responsible
for creating them or coordinating the functionality between them. The
Query class has several methods that are important for
createWeight(IndexSearcher searcher, ScoreMode scoreMode, float boost)— A
Weightis the internal representation of the Query, so each Query implementation must provide an implementation of Weight. See the subsection on The Weight Interface below for details on implementing the Weight interface.
rewrite(IndexReader reader)— Rewrites queries into primitive queries. Primitive queries are:
BooleanQuery, and other queries that implement
createWeight(IndexSearcher searcher,ScoreMode scoreMode, float boost)
The Weight Interface
interface provides an internal representation of the Query so that it can be reused. Any
dependent state should be stored in the Weight implementation,
not in the Query class. The interface defines four main methods:
scorer()— Construct a new
Scorerfor this Weight. See The Scorer Class below for help defining a Scorer. As the name implies, the Scorer is responsible for doing the actual scoring of documents given the Query.
explain(LeafReaderContext context, int doc)— Provide a means for explaining why a given document was scored the way it was. Typically a weight such as TermWeight that scores via a
Similaritywill make use of the Similarity's implementation:
SimScorer#explain(Explanation freq, long norm).
extractTerms(Set<Term> terms)— Extract terms that this query operates on. This is typically used to support distributed search: knowing the terms that a query operates on helps merge index statistics of these terms so that scores are computed over a subset of the data like they would if all documents were in the same index.
matches(LeafReaderContext context, int doc)— Give information about positions and offsets of matches. This is typically useful to implement highlighting.
The Scorer Class
abstract class provides common scoring functionality for all Scorer implementations and
is the heart of the Lucene scoring process. The Scorer defines the following methods which
must be implemented:
iterator()— Return a
DocIdSetIteratorthat can iterate over all document that matches this Query.
docID()— Returns the id of the
Documentthat contains the match.
score()— Return the score of the current document. This value can be determined in any appropriate way for an application. For instance, the
TermScorersimply defers to the configured Similarity:
SimScorer.score(float freq, long norm).
getChildren()— Returns any child subscorers underneath this scorer. This allows for users to navigate the scorer hierarchy and receive more fine-grained details on the scoring process.
The BulkScorer Class
BulkScorer scores a range of documents. There is only one
score(LeafCollector,Bits,int,int)— Score all documents up to but not including the specified max document.
Why would I want to add my own Query?
In a nutshell, you want to add your own custom Query implementation when you think that Lucene's aren't appropriate for the task that you want to do. You might be doing some cutting edge research or you need more information back out of Lucene (similar to Doug adding SpanQuery functionality).
Appendix: Search Algorithm
This section is mostly notes on stepping through the Scoring process and serves as fertilizer for the earlier sections.
Once inside the IndexSearcher, a
is used for the scoring and sorting of the search results.
These important objects are involved in a search:
Weightobject of the Query. The Weight object is an internal representation of the Query that allows the Query to be reused by the IndexSearcher.
- The IndexSearcher that initiated the call.
Sortobject for specifying how to sort the results if the standard score-based sort method is not desired.
Assuming we are not sorting (since sorting doesn't affect the raw Lucene score),
we call one of the search methods of the IndexSearcher, passing in the
Weight object created by
IndexSearcher.createWeight(Query,ScoreMode,float) and the number of results we want.
This method returns a
which is an internal collection of search results. The IndexSearcher creates
passes it along with the Weight to another expert search method (for
more on the
IndexSearcher). The TopScoreDocCollector
PriorityQueue to collect the
top results for the search.
At last, we are actually going to score some documents. The score method takes in the Collector
(most likely the TopScoreDocCollector or TopFieldCollector) and does its business. Of course, here
is where things get involved. The
Scorer that is returned
Weight object depends on what type of Query was
submitted. In most real world applications with multiple query terms, the
Scorer is going to be a
BooleanWeight (see the section on
custom queries for info on changing this).
Assuming a BooleanScorer2, we get a internal Scorer based on the required, optional and prohibited parts of the query.
Using this internal Scorer, the BooleanScorer2 then proceeds into a while loop based on the
DocIdSetIterator.nextDoc() method. The nextDoc() method advances
to the next document matching the query. This is an abstract method in the Scorer class and is thus
overridden by all derived implementations. If you have a simple OR query your internal Scorer is most
likely a DisjunctionSumScorer, which essentially combines the scorers from the sub scorers of the OR'd terms.
Interface Summary Interface Description BoostAttributeAdd this
MultiTermQuery.getTermsEnum(Terms,AttributeSource)and update the boost on each returned term.
CollectorExpert: Collectors are primarily meant to be used to gather raw results from a search, and implement sorting or custom result filtering, collation, etc. CollectorManager<C extends Collector,T>A manager of collectors. LeafCollectorCollector decouples the score from the collected doc: the score computation is skipped entirely if it's not needed. LeafFieldComparatorExpert: comparator that gets instantiated on each leaf from a top-level
Matches MatchesIteratorAn iterator over match positions (and optionally offsets) for a single document and field To iterate over the matches, call
MatchesIterator.next()until it returns
false, retrieving positions and/or offsets after each call.
Attributeto a fresh
QueryCacheA cache for queries. QueryCachingPolicyA policy defining which filters should be cached. ReferenceManager.RefreshListenerUse to receive notification when a refresh has finished. SearcherLifetimeManager.Pruner SegmentCacheableInterface defining whether or not an object can be cached against a
LeafReaderObjects that depend only on segment-immutable structures such as Points or postings lists can just return
SegmentCacheable.isCacheable(LeafReaderContext)Objects that depend on doc values should return
DocValues.isCacheable(LeafReaderContext, String...), which will check to see if the doc values fields have been updated.
Class Summary Class Description AutomatonQueryA
Querythat will match terms against a finite-state machine.
Querythat blends index statistics across multiple terms.
BlendedTermQuery.BuilderA Builder for
BlendedTermQuery.RewriteMethoddefines how queries for individual terms should be merged.
BlockMaxDISI BooleanClauseA clause in a BooleanQuery. BooleanQueryA Query that matches documents matching boolean combinations of other queries, e.g. BooleanQuery.BuilderA builder for boolean queries. BoostAttributeImplImplementation class for
Querywrapper that allows to give a boost to the wrapped query.
BulkScorerThis class is used to score a range of documents at once, and is returned by
CachingCollectorCaches all docs, and optionally also scores, coming from a search, and is then able to replay them to another collector. CollectionStatisticsContains statistics for a collection (field). ConjunctionDISIA conjunction of DocIdSetIterators. ConstantScoreQueryA query that wraps another query and simply returns a constant score equal to 1 for every document that matches the query. ConstantScoreQuery.ConstantBulkScorerWe return this as our
BulkScorerso that if the CSQ wraps a query with its own optimized top-level scorer (e.g.
ConstantScoreWeightA Weight that has a constant score equal to the boost of the wrapped query. ControlledRealTimeReopenThread<T>Utility class that runs a thread to manage periodicc reopens of a
ReferenceManager, with methods to wait for a specific index changes to become visible.
DisiPriorityQueueA priority queue of DocIdSetIterators that orders by current doc ID. DisiWrapperWrapper used in
DocIdSetIteratorwhich is a disjunction of the approximations of the provided iterators.
DisjunctionMaxQueryA query that generates the union of documents produced by its subqueries, and that scores each document with the maximum score for that document as produced by any subquery, plus a tie breaking increment for any additional matching subqueries. DocIdSetA DocIdSet contains a set of doc ids. DocIdSetIteratorThis abstract class defines methods to iterate over a set of non-decreasing doc ids. DocValuesFieldExistsQueryA
Querythat matches documents that have a value for a given field as reported by doc values iterators.
DocValuesRewriteMethodRewrites MultiTermQueries into a filter, using DocValues for term enumeration. DoubleValuesPer-segment, per-document double values, which can be calculated at search-time DoubleValuesSourceBase class for producing
DoubleValuesTo obtain a
DoubleValuesobject for a leaf reader, clients should call
DoubleValuesSource.rewrite(IndexSearcher)against the top-level searcher, and then call
DoubleValuesSource.getValues(LeafReaderContext, DoubleValues)on the resulting DoubleValuesSource.
ExplanationExpert: Describes the score computation for document and query. FieldComparator<T>Expert: a FieldComparator compares hits so as to determine their sort order when collecting the top results with
FieldComparator.DocComparatorSorts by ascending docID FieldComparator.DoubleComparatorParses field's values as double (using
LeafReader.getNumericDocValues(java.lang.String)and sorts by ascending value
FieldComparator.FloatComparatorParses field's values as float (using
LeafReader.getNumericDocValues(String)and sorts by ascending value
FieldComparator.IntComparatorParses field's values as int (using
LeafReader.getNumericDocValues(String)and sorts by ascending value
FieldComparator.LongComparatorParses field's values as long (using
LeafReader.getNumericDocValues(String)and sorts by ascending value
FieldComparator.NumericComparator<T extends Number>Base FieldComparator class for numeric types FieldComparator.RelevanceComparatorSorts by descending relevance. FieldComparator.TermOrdValComparatorSorts by field's natural Term sort order, using ordinals. FieldComparator.TermValComparatorSorts by field's natural Term sort order. FieldComparatorSourceProvides a
FieldComparatorfor custom field sorting.
FieldDocExpert: A ScoreDoc which also contains information about how to sort the referenced document. FieldValueHitQueue<T extends FieldValueHitQueue.Entry>Expert: A hit queue for sorting by hits by terms in more than one field. FieldValueHitQueue.EntryExtension of ScoreDoc to also store the
FilteredDocIdSetIteratorAbstract decorator class of a DocIdSetIterator implementation that provides on-demand filter/validation mechanism on an underlying DocIdSetIterator. FilterLeafCollector
FilterMatchesIteratorA MatchesIterator that delegates all calls to another MatchesIterator FilterScorable FilterScorerA
Scorer, which it uses as its basic source of data, possibly transforming the data along the way or providing additional functionality.
Weightand implements all abstract methods by calling the contained weight's method.
FuzzyQueryImplements the fuzzy search query. FuzzyTermsEnumSubclass of TermsEnum for enumerating all terms that are similar to the specified filter term. ImpactsDISI
DocIdSetIteratorthat skips non-competitive docs thanks to the indexed impacts.
IndexOrDocValuesQueryA query that uses either an index structure (points or terms) or doc values in order to run a query, depending which one is more efficient. IndexSearcherImplements search over a single IndexReader. IndexSearcher.LeafSliceA class holding a subset of the
IndexSearchers leaf contexts to be executed within a single thread.
LeafSimScorer LiveFieldValues<S,T>Tracks live field values across NRT reader reopens. LongValuesPer-segment, per-document long values, which can be calculated at search-time LongValuesSourceBase class for producing
LongValuesTo obtain a
LongValuesobject for a leaf reader, clients should call
LongValuesSource.rewrite(IndexSearcher)against the top-level searcher, and then
QueryCachethat evicts queries using a LRU (least-recently-used) eviction policy in order to remain under a given maximum size and number of bytes used.
MatchAllDocsQueryA query that matches all documents. MatchesUtils MatchNoDocsQueryA query that matches no documents. MaxNonCompetitiveBoostAttributeImplImplementation class for
MultiCollector MultiCollectorManager MultiPhraseQueryA generalized version of
PhraseQuery, with the possibility of adding more than one term at the same position that are treated as a disjunction (OR).
MultiPhraseQuery.BuilderA builder for multi-phrase queries MultiTermQuery MultiTermQuery.RewriteMethodAbstract class that defines how the query is rewritten. MultiTermQuery.TopTermsBlendedFreqScoringRewriteA rewrite method that first translates each term into
BooleanClause.Occur.SHOULDclause in a BooleanQuery, but adjusts the frequencies used for scoring to be blended across the terms, otherwise the rarest term typically ranks highest (often not useful eg in the set of expanded terms in a FuzzyQuery).
MultiTermQuery.TopTermsBoostOnlyBooleanQueryRewriteA rewrite method that first translates each term into
BooleanClause.Occur.SHOULDclause in a BooleanQuery, but the scores are only computed as the boost.
MultiTermQuery.TopTermsScoringBooleanQueryRewriteA rewrite method that first translates each term into
BooleanClause.Occur.SHOULDclause in a BooleanQuery, and keeps the scores as computed by the query.
NamedMatchesUtility class to help extract the set of sub queries that have matched from a larger query. NGramPhraseQueryThis is a
PhraseQuerywhich is optimized for n-gram phrase query.
Querythat matches documents that have a value for a given field as reported by field norms.
PhraseQueryA Query that matches documents containing a particular sequence of terms. PhraseQuery.BuilderA builder for phrase queries. PointInSetQueryAbstract query class to find all documents whose single or multi-dimensional point values, previously indexed with e.g. PointInSetQuery.StreamIterator of encoded point values. PointRangeQueryAbstract class for range queries against single or multidimensional points such as
PositiveScoresOnlyCollector PrefixQueryA Query that matches documents containing terms with a specified prefix. QueryThe abstract base class for queries. QueryRescorerA
Rescorerthat uses a provided Query to assign scores to the first-pass hits.
QueryVisitorAllows recursion through a query tree ReferenceManager<G>Utility class to safely share instances of a certain type across multiple threads, while periodically refreshing them. RegexpQueryA fast regular expression query based on the
RescorerRe-scores the topN results (
TopDocs) from an original query.
ScorableAllows access to the score of a Query Scorable.ChildScorableA child Scorer and its relationship to its parent. ScoreCachingWrappingScorerA
Scorerwhich wraps another scorer and caches the score of the current document.
ScoreDocHolds one hit in
ScorerExpert: Common scoring functionality for different types of queries. ScorerSupplierA supplier of
ScoringRewrite<B>Base rewrite method that translates each term into a query, and keeps the scores as computed by the query. SearcherFactoryFactory class used by
SearcherManagerto create new IndexSearchers.
SearcherLifetimeManagerKeeps track of current plus old IndexSearchers, closing the old ones once they have timed out. SearcherLifetimeManager.PruneByAgeSimple pruner that drops any searcher older by more than the specified seconds, than the newest searcher. SearcherManagerUtility class to safely share
IndexSearcherinstances across multiple threads, while periodically reopening.
Collectorimplementation that is used to collect all contexts.
FieldComparatorimplementation that is used for all contexts.
SortEncapsulates sort criteria for returned hits. SortedNumericSelectorSelects a value from the document's list to use as the representative value SortedNumericSortFieldSortField for
SortedSetSelectorSelects a value from the document's set to use as the representative value SortedSetSortFieldSortField for
SortFieldStores information about how to sort documents by terms in an individual field. SortRescorerA
Rescorerthat re-sorts according to a provided Sort.
SynonymQueryA query that treats multiple terms as synonyms. SynonymQuery.BuilderA builder for
TermInSetQuery TermQueryA Query that matches documents containing a term. TermRangeQueryA Query that matches documents within an range of terms. TermStatisticsContains statistics for a specific term TimeLimitingCollectorThe
TimeLimitingCollectoris used to timeout search requests that take longer than the maximum allowed search time limit.
TimeLimitingCollector.TimerThreadThread used to timeout search requests. TopDocsRepresents hits returned by
TopDocsCollector<T extends ScoreDoc>A base class for all collectors that return a
TopFieldCollector TopFieldDocsRepresents hits returned by
TopScoreDocCollector TopTermsRewrite<B>Base rewrite method for collecting only the top terms via a priority queue. TotalHitCountCollectorJust counts the total number of hits. TotalHitsDescription of the total number of hits of a query. TwoPhaseIterator UsageTrackingQueryCachingPolicyA
QueryCachingPolicythat tracks usage statistics of recently-used filters in order to decide on which filters are worth caching.
WeightExpert: Calculate query weights and build query scorers. Weight.DefaultBulkScorerJust wraps a Scorer and performs top scoring using it. WildcardQueryImplements the wildcard search query.
Enum Summary Enum Description BooleanClause.OccurSpecifies how clauses are to occur in matching documents. ScoreModeDifferent modes of search. SortedNumericSelector.TypeType of selection to perform. SortedSetSelector.TypeType of selection to perform. SortField.TypeSpecifies the type of the terms to be sorted, or special types such as CUSTOM TotalHits.RelationHow the
TotalHits.valueshould be interpreted.
Exception Summary Exception Description BooleanQuery.TooManyClausesThrown when an attempt is made to add more than
CollectionTerminatedExceptionThrow this exception in
LeafCollector.collect(int)to prematurely terminate collection of the current leaf.
FuzzyTermsEnum.FuzzyTermsExceptionThrown to indicate that there was an issue creating a fuzzy query for a given term. TimeLimitingCollector.TimeExceededExceptionThrown when elapsed search time exceeds allowed search time.