Code to search indices.


Interface Summary
BoostAttribute Add this Attribute to a TermsEnum returned by MultiTermQuery.getTermsEnum(Terms,AttributeSource) and update the boost on each returned term.
FieldCache Expert: Maintains caches of term values.
FieldCache.ByteParser Deprecated.
FieldCache.DoubleParser Interface to parse doubles from document fields.
FieldCache.FloatParser Interface to parse floats from document fields.
FieldCache.IntParser Interface to parse ints from document fields.
FieldCache.LongParser Interface to parse long from document fields.
FieldCache.Parser Marker interface as super-interface to all parsers.
FieldCache.ShortParser Deprecated.
FuzzyTermsEnum.LevenshteinAutomataAttribute reuses compiled automata across different segments, because they are independent of the index
MaxNonCompetitiveBoostAttribute Add this Attribute to a fresh AttributeSource before calling MultiTermQuery.getTermsEnum(Terms,AttributeSource).
ReferenceManager.RefreshListener Use to receive notification when a refresh has finished.
SearcherLifetimeManager.Pruner See SearcherLifetimeManager.prune(

Class Summary
AutomatonQuery A Query that will match terms against a finite-state machine.
BitsFilteredDocIdSet This implementation supplies a filtered DocIdSet, that excludes all docids which are not in a Bits instance.
BooleanClause A clause in a BooleanQuery.
BooleanQuery A Query that matches documents matching boolean combinations of other queries, e.g.
BoostAttributeImpl Implementation class for BoostAttribute.
CachingCollector Caches all docs, and optionally also scores, coming from a search, and is then able to replay them to another collector.
CachingWrapperFilter Wraps another Filter's result and caches it.
CollectionStatistics Contains statistics for a collection (field)
Collector Expert: Collectors are primarily meant to be used to gather raw results from a search, and implement sorting or custom result filtering, collation, etc.
ComplexExplanation Expert: Describes the score computation for document and query, and can distinguish a match independent of a positive value.
ConstantScoreQuery A query that wraps another query or a filter and simply returns a constant score equal to the query boost for every document that matches the filter or query.
ControlledRealTimeReopenThread<T> Utility class that runs a thread to manage periodicc reopens of a ReferenceManager, with methods to wait for a specific index changes to become visible.
DisjunctionMaxQuery A query that generates the union of documents produced by its subqueries, and that scores each document with the maximum score for that document as produced by any subquery, plus a tie breaking increment for any additional matching subqueries.
DocIdSet A DocIdSet contains a set of doc ids.
DocIdSetIterator This abstract class defines methods to iterate over a set of non-decreasing doc ids.
DocTermOrdsRangeFilter A range filter built on top of a cached multi-valued term field (in FieldCache).
DocTermOrdsRewriteMethod Rewrites MultiTermQueries into a filter, using DocTermOrds for term enumeration.
Explanation Expert: Describes the score computation for document and query.
FieldCache.Bytes Field values as 8-bit signed bytes
FieldCache.CacheEntry EXPERT: A unique Identifier/Description for each item in the FieldCache.
FieldCache.CreationPlaceholder Placeholder indicating creation of this cache is currently in-progress.
FieldCache.Doubles Field values as 64-bit doubles
FieldCache.Floats Field values as 32-bit floats
FieldCache.Ints Field values as 32-bit signed integers
FieldCache.Longs Field values as 64-bit signed long integers
FieldCache.Shorts Field values as 16-bit signed shorts
FieldCacheDocIdSet Base class for DocIdSet to be used with FieldCache.
FieldCacheRangeFilter<T> A range filter built on top of a cached single term field (in FieldCache).
FieldCacheRewriteMethod Rewrites MultiTermQueries into a filter, using the FieldCache for term enumeration.
FieldCacheTermsFilter A Filter that only accepts documents whose single term value in the specified field is contained in the provided set of allowed terms.
FieldComparator<T> Expert: a FieldComparator compares hits so as to determine their sort order when collecting the top results with TopFieldCollector.
FieldComparator.ByteComparator Deprecated.
FieldComparator.DocComparator Sorts by ascending docID
FieldComparator.DoubleComparator Parses field's values as double (using FieldCache.getDoubles(org.apache.lucene.index.AtomicReader, java.lang.String, boolean) and sorts by ascending value
FieldComparator.FloatComparator Parses field's values as float (using FieldCache.getFloats(org.apache.lucene.index.AtomicReader, java.lang.String, boolean) and sorts by ascending value
FieldComparator.IntComparator Parses field's values as int (using FieldCache.getInts(org.apache.lucene.index.AtomicReader, java.lang.String, boolean) and sorts by ascending value
FieldComparator.LongComparator Parses field's values as long (using FieldCache.getLongs(org.apache.lucene.index.AtomicReader, java.lang.String, boolean) and sorts by ascending value
FieldComparator.NumericComparator<T extends Number> Base FieldComparator class for numeric types
FieldComparator.RelevanceComparator Sorts by descending relevance.
FieldComparator.ShortComparator Deprecated.
FieldComparator.TermOrdValComparator Sorts by field's natural Term sort order, using ordinals.
FieldComparator.TermValComparator Sorts by field's natural Term sort order.
FieldComparatorSource Provides a FieldComparator for custom field sorting.
FieldDoc Expert: A ScoreDoc which also contains information about how to sort the referenced document.
FieldValueFilter A Filter that accepts all documents that have one or more values in a given field.
FieldValueHitQueue<T extends FieldValueHitQueue.Entry> Expert: A hit queue for sorting by hits by terms in more than one field.
FieldValueHitQueue.Entry Extension of ScoreDoc to also store the FieldComparator slot.
Filter Abstract base class for restricting which documents may be returned during searching.
FilteredDocIdSet Abstract decorator class for a DocIdSet implementation that provides on-demand filtering/validation mechanism on a given DocIdSet.
FilteredDocIdSetIterator Abstract decorator class of a DocIdSetIterator implementation that provides on-demand filter/validation mechanism on an underlying DocIdSetIterator.
FilteredQuery A query that applies a filter to the results of another query.
FilteredQuery.FilterStrategy Abstract class that defines how the filter (DocIdSet) applied during document collection.
FilteredQuery.RandomAccessFilterStrategy A FilteredQuery.FilterStrategy that conditionally uses a random access filter if the given DocIdSet supports random access (returns a non-null value from DocIdSet.bits()) and FilteredQuery.RandomAccessFilterStrategy.useRandomAccess(Bits, int) returns true.
FuzzyQuery Implements the fuzzy search query.
FuzzyTermsEnum Subclass of TermsEnum for enumerating all terms that are similar to the specified filter term.
FuzzyTermsEnum.LevenshteinAutomataAttributeImpl Stores compiled automata as a list (indexed by edit distance)
IndexSearcher Implements search over a single IndexReader.
IndexSearcher.LeafSlice A class holding a subset of the IndexSearchers leaf contexts to be executed within a single thread.
LiveFieldValues<S,T> Tracks live field values across NRT reader reopens.
MatchAllDocsQuery A query that matches all documents.
MaxNonCompetitiveBoostAttributeImpl Implementation class for MaxNonCompetitiveBoostAttribute.
MultiCollector A Collector which allows running a search with several Collectors.
MultiPhraseQuery MultiPhraseQuery is a generalized version of PhraseQuery, with an added method MultiPhraseQuery.add(Term[]).
MultiTermQuery An abstract Query that matches documents containing a subset of terms provided by a FilteredTermsEnum enumeration.
MultiTermQuery.ConstantScoreAutoRewrite A rewrite method that tries to pick the best constant-score rewrite method based on term and document counts from the query.
MultiTermQuery.RewriteMethod Abstract class that defines how the query is rewritten.
MultiTermQuery.TopTermsBoostOnlyBooleanQueryRewrite A rewrite method that first translates each term into BooleanClause.Occur.SHOULD clause in a BooleanQuery, but the scores are only computed as the boost.
MultiTermQuery.TopTermsScoringBooleanQueryRewrite A rewrite method that first translates each term into BooleanClause.Occur.SHOULD clause in a BooleanQuery, and keeps the scores as computed by the query.
MultiTermQueryWrapperFilter<Q extends MultiTermQuery> A wrapper for MultiTermQuery, that exposes its functionality as a Filter.
NGramPhraseQuery This is a PhraseQuery which is optimized for n-gram phrase query.
NumericRangeFilter<T extends Number> A Filter that only accepts numeric values within a specified range.
NumericRangeQuery<T extends Number> A Query that matches numeric values within a specified range.
PhraseQuery A Query that matches documents containing a particular sequence of terms.
PositiveScoresOnlyCollector A Collector implementation which wraps another Collector and makes sure only documents with scores > 0 are collected.
PrefixFilter A Filter that restricts search results to values that have a matching prefix in a given field.
PrefixQuery A Query that matches documents containing terms with a specified prefix.
PrefixTermsEnum Subclass of FilteredTermEnum for enumerating all terms that match the specified prefix filter term.
Query The abstract base class for queries.
QueryWrapperFilter Constrains search results to only match those which also match a provided query.
ReferenceManager<G> Utility class to safely share instances of a certain type across multiple threads, while periodically refreshing them.
RegexpQuery A fast regular expression query based on the org.apache.lucene.util.automaton package.
ScoreCachingWrappingScorer A Scorer which wraps another scorer and caches the score of the current document.
ScoreDoc Holds one hit in TopDocs.
Scorer Expert: Common scoring functionality for different types of queries.
Scorer.ChildScorer A child Scorer and its relationship to its parent.
ScoringRewrite<Q extends Query> Base rewrite method that translates each term into a query, and keeps the scores as computed by the query.
SearcherFactory Factory class used by SearcherManager to create new IndexSearchers.
SearcherLifetimeManager Keeps track of current plus old IndexSearchers, closing the old ones once they have timed out.
SearcherLifetimeManager.PruneByAge Simple pruner that drops any searcher older by more than the specified seconds, than the newest searcher.
SearcherManager Utility class to safely share IndexSearcher instances across multiple threads, while periodically reopening.
Sort Encapsulates sort criteria for returned hits.
SortField Stores information about how to sort documents by terms in an individual field.
TermQuery A Query that matches documents containing a term.
TermRangeFilter A Filter that restricts search results to a range of term values in a given field.
TermRangeQuery A Query that matches documents within an range of terms.
TermRangeTermsEnum Subclass of FilteredTermEnum for enumerating all terms that match the specified range parameters.
TermStatistics Contains statistics for a specific term
TimeLimitingCollector The TimeLimitingCollector is used to timeout search requests that take longer than the maximum allowed search time limit.
TimeLimitingCollector.TimerThread Thread used to timeout search requests.
TopDocs Represents hits returned by,Filter,int) and,int).
TopDocsCollector<T extends ScoreDoc> A base class for all collectors that return a TopDocs output.
TopFieldCollector A Collector that sorts by SortField using FieldComparators.
TopFieldDocs Represents hits returned by,Filter,int,Sort).
TopScoreDocCollector A Collector implementation that collects the top-scoring hits, returning them as a TopDocs.
TopTermsRewrite<Q extends Query> Base rewrite method for collecting only the top terms via a priority queue.
TotalHitCountCollector Just counts the total number of hits.
Weight Expert: Calculate query weights and build query scorers.
WildcardQuery Implements the wildcard search query.

Enum Summary
BooleanClause.Occur Specifies how clauses are to occur in matching documents.
SortField.Type Specifies the type of the terms to be sorted, or special types such as CUSTOM

Exception Summary
BooleanQuery.TooManyClauses Thrown when an attempt is made to add more than BooleanQuery.getMaxClauseCount() clauses.
CollectionTerminatedException Throw this exception in Collector.collect(int) to prematurely terminate collection of the current leaf.
TimeLimitingCollector.TimeExceededException Thrown when elapsed search time exceeds allowed search time.

Package Description

Code to search indices.

Table Of Contents

  1. Search Basics
  2. The Query Classes
  3. Scoring: Introduction
  4. Scoring: Basics
  5. Changing the Scoring
  6. Appendix: Search Algorithm

Search Basics

Lucene offers a wide variety of Query implementations, most of which are in this package, its subpackages (spans, payloads), or the queries module. These implementations can be combined in a wide variety of ways to provide complex querying capabilities along with information about where matches took place in the document collection. The Query Classes section below highlights some of the more important Query classes. For details on implementing your own Query class, see Custom Queries -- Expert Level below.

To perform a search, applications usually call,int) or,Filter,int).

Once a Query has been created and submitted to the IndexSearcher, the scoring process begins. After some infrastructure setup, control finally passes to the Weight implementation and its Scorer instances. See the Algorithm section for more notes on the process.

Query Classes


Of the various implementations of Query, the TermQuery is the easiest to understand and the most often used in applications. A TermQuery matches all the documents that contain the specified Term, which is a word that occurs in a certain Field. Thus, a TermQuery identifies and scores all Documents that have a Field with the specified string in it. Constructing a TermQuery is as simple as:

        TermQuery tq = new TermQuery(new Term("fieldName", "term"));
In this example, the Query identifies all Documents that have the Field named "fieldName" containing the word "term".


Things start to get interesting when one combines multiple TermQuery instances into a BooleanQuery. A BooleanQuery contains multiple BooleanClauses, where each clause contains a sub-query (Query instance) and an operator (from BooleanClause.Occur) describing how that sub-query is combined with the other clauses:

  1. SHOULD — Use this operator when a clause can occur in the result set, but is not required. If a query is made up of all SHOULD clauses, then every document in the result set matches at least one of these clauses.

  2. MUST — Use this operator when a clause is required to occur in the result set. Every document in the result set will match all such clauses.

  3. MUST NOT — Use this operator when a clause must not occur in the result set. No document in the result set will match any such clauses.

Boolean queries are constructed by adding two or more BooleanClause instances. If too many clauses are added, a TooManyClauses exception will be thrown during searching. This most often occurs when a Query is rewritten into a BooleanQuery with many TermQuery clauses, for example by WildcardQuery. The default setting for the maximum number of clauses 1024, but this can be changed via the static method BooleanQuery.setMaxClauseCount(int).


Another common search is to find documents containing certain phrases. This is handled three different ways:

  1. PhraseQuery — Matches a sequence of Terms. PhraseQuery uses a slop factor to determine how many positions may occur between any two terms in the phrase and still be considered a match. The slop is 0 by default, meaning the phrase must match exactly.

  2. MultiPhraseQuery — A more general form of PhraseQuery that accepts multiple Terms for a position in the phrase. For example, this can be used to perform phrase queries that also incorporate synonyms.

  3. SpanNearQuery — Matches a sequence of other SpanQuery instances. SpanNearQuery allows for much more complicated phrase queries since it is constructed from other SpanQuery instances, instead of only TermQuery instances.


The TermRangeQuery matches all documents that occur in the exclusive range of a lower Term and an upper Term according to TermsEnum.getComparator(). It is not intended for numerical ranges; use NumericRangeQuery instead. For example, one could find all documents that have terms beginning with the letters a through c.


The NumericRangeQuery matches all documents that occur in a numeric range. For NumericRangeQuery to work, you must index the values using a one of the numeric fields (IntField, LongField, FloatField, or DoubleField).

PrefixQuery, WildcardQuery, RegexpQuery

While the PrefixQuery has a different implementation, it is essentially a special case of the WildcardQuery. The PrefixQuery allows an application to identify all documents with terms that begin with a certain string. The WildcardQuery generalizes this by allowing for the use of * (matches 0 or more characters) and ? (matches exactly one character) wildcards. Note that the WildcardQuery can be quite slow. Also note that WildcardQuery should not start with * and ?, as these are extremely slow. Some QueryParsers may not allow this by default, but provide a setAllowLeadingWildcard method to remove that protection. The RegexpQuery is even more general than WildcardQuery, allowing an application to identify all documents with terms that match a regular expression pattern.


A FuzzyQuery matches documents that contain terms similar to the specified term. Similarity is determined using Levenshtein (edit) distance. This type of query can be useful when accounting for spelling variations in the collection.

Scoring — Introduction

Lucene scoring is the heart of why we all love Lucene. It is blazingly fast and it hides almost all of the complexity from the user. In a nutshell, it works. At least, that is, until it doesn't work, or doesn't work as one would expect it to work. Then we are left digging into Lucene internals or asking for help on to figure out why a document with five of our query terms scores lower than a different document with only one of the query terms.

While this document won't answer your specific scoring issues, it will, hopefully, point you to the places that can help you figure out the what and why of Lucene scoring.

Lucene scoring supports a number of pluggable information retrieval models, including:

These models can be plugged in via the Similarity API, and offer extension hooks and parameters for tuning. In general, Lucene first finds the documents that need to be scored based on boolean logic in the Query specification, and then ranks this subset of matching documents via the retrieval model. For some valuable references on VSM and IR in general refer to Lucene Wiki IR references.

The rest of this document will cover Scoring basics and explain how to change your Similarity. Next, it will cover ways you can customize the lucene internals in Custom Queries -- Expert Level, which gives details on implementing your own Query class and related functionality. Finally, we will finish up with some reference material in the Appendix.

Scoring — Basics

Scoring is very much dependent on the way documents are indexed, so it is important to understand indexing. (see Lucene overview before continuing on with this section) Be sure to use the useful IndexSearcher.explain(Query, doc) to understand how the score for a certain matching document was computed.

Generally, the Query determines which documents match (a binary decision), while the Similarity determines how to assign scores to the matching documents.

Fields and Documents

In Lucene, the objects we are scoring are Documents. A Document is a collection of Fields. Each Field has semantics about how it is created and stored (tokenized, stored, etc). It is important to note that Lucene scoring works on Fields and then combines the results to return Documents. This is important because two Documents with the exact same content, but one having the content in two Fields and the other in one Field may return different scores for the same query due to length normalization.

Score Boosting

Lucene allows influencing search results by "boosting" at different times:

Indexing time boosts are pre-processed for storage efficiency and written to storage for a field as follows:

Changing Scoring — Similarity

Changing Similarity is an easy way to influence scoring, this is done at index-time with IndexWriterConfig.setSimilarity(Similarity) and at query-time with IndexSearcher.setSimilarity(Similarity). Be sure to use the same Similarity at query-time as at index-time (so that norms are encoded/decoded correctly); Lucene makes no effort to verify this.

You can influence scoring by configuring a different built-in Similarity implementation, or by tweaking its parameters, subclassing it to override behavior. Some implementations also offer a modular API which you can extend by plugging in a different component (e.g. term frequency normalizer).

Finally, you can extend the low level Similarity directly to implement a new retrieval model, or to use external scoring factors particular to your application. For example, a custom Similarity can access per-document values via FieldCache or NumericDocValues and integrate them into the score.

See the package documentation for information on the built-in available scoring models and extending or changing Similarity.

Custom Queries — Expert Level

Custom queries are an expert level task, so tread carefully and be prepared to share your code if you want help.

With the warning out of the way, it is possible to change a lot more than just the Similarity when it comes to matching and scoring in Lucene. Lucene's search is a complex mechanism that is grounded by three main classes:

  1. Query — The abstract object representation of the user's information need.
  2. Weight — The internal interface representation of the user's Query, so that Query objects may be reused. This is global (across all segments of the index) and generally will require global statistics (such as docFreq for a given term across all segments).
  3. Scorer — An abstract class containing common functionality for scoring. Provides both scoring and explanation capabilities. This is created per-segment.
Details on each of these classes, and their children, can be found in the subsections below.

The Query Class

In some sense, the Query class is where it all begins. Without a Query, there would be nothing to score. Furthermore, the Query class is the catalyst for the other scoring classes as it is often responsible for creating them or coordinating the functionality between them. The Query class has several methods that are important for derived classes:

  1. createWeight(IndexSearcher searcher) — A Weight is the internal representation of the Query, so each Query implementation must provide an implementation of Weight. See the subsection on The Weight Interface below for details on implementing the Weight interface.
  2. rewrite(IndexReader reader) — Rewrites queries into primitive queries. Primitive queries are: TermQuery, BooleanQuery, and other queries that implement createWeight(IndexSearcher searcher)

The Weight Interface

The Weight interface provides an internal representation of the Query so that it can be reused. Any IndexSearcher dependent state should be stored in the Weight implementation, not in the Query class. The interface defines five methods that must be implemented:

  1. getQuery() — Pointer to the Query that this Weight represents.
  2. getValueForNormalization() — A weight can return a floating point value to indicate its magnitude for query normalization. Typically a weight such as TermWeight that scores via a Similarity will just defer to the Similarity's implementation: SimWeight#getValueForNormalization(). For example, with Lucene's classic vector-space formula, this is implemented as the sum of squared weights: (idf * boost)2
  3. normalize(float norm, float topLevelBoost) — Performs query normalization: Typically a weight such as TermWeight that scores via a Similarity will just defer to the Similarity's implementation: SimWeight#normalize(float,float).
  4. scorer(AtomicReaderContext context, boolean scoresDocsInOrder, boolean topScorer, Bits acceptDocs) — Construct a new Scorer for this Weight. See The Scorer Class below for help defining a Scorer. As the name implies, the Scorer is responsible for doing the actual scoring of documents given the Query.
  5. explain(AtomicReaderContext context, int doc) — Provide a means for explaining why a given document was scored the way it was. Typically a weight such as TermWeight that scores via a Similarity will make use of the Similarity's implementation: SimScorer#explain(int doc, Explanation freq).

The Scorer Class

The Scorer abstract class provides common scoring functionality for all Scorer implementations and is the heart of the Lucene scoring process. The Scorer defines the following abstract (some of them are not yet abstract, but will be in future versions and should be considered as such now) methods which must be implemented (some of them inherited from DocIdSetIterator):

  1. nextDoc() — Advances to the next document that matches this Query, returning true if and only if there is another document that matches.
  2. docID() — Returns the id of the Document that contains the match.
  3. score() — Return the score of the current document. This value can be determined in any appropriate way for an application. For instance, the TermScorer simply defers to the configured Similarity: SimScorer.score(int doc, float freq).
  4. freq() — Returns the number of matches for the current document. This value can be determined in any appropriate way for an application. For instance, the TermScorer simply defers to the term frequency from the inverted index: DocsEnum.freq().
  5. advance() — Skip ahead in the document matches to the document whose id is greater than or equal to the passed in value. In many instances, advance can be implemented more efficiently than simply looping through all the matching documents until the target document is identified.
  6. getChildren() — Returns any child subscorers underneath this scorer. This allows for users to navigate the scorer hierarchy and receive more fine-grained details on the scoring process.

Why would I want to add my own Query?

In a nutshell, you want to add your own custom Query implementation when you think that Lucene's aren't appropriate for the task that you want to do. You might be doing some cutting edge research or you need more information back out of Lucene (similar to Doug adding SpanQuery functionality).

Appendix: Search Algorithm

This section is mostly notes on stepping through the Scoring process and serves as fertilizer for the earlier sections.

In the typical search application, a Query is passed to the IndexSearcher, beginning the scoring process.

Once inside the IndexSearcher, a Collector is used for the scoring and sorting of the search results. These important objects are involved in a search:

  1. The Weight object of the Query. The Weight object is an internal representation of the Query that allows the Query to be reused by the IndexSearcher.
  2. The IndexSearcher that initiated the call.
  3. A Filter for limiting the result set. Note, the Filter may be null.
  4. A Sort object for specifying how to sort the results if the standard score-based sort method is not desired.

Assuming we are not sorting (since sorting doesn't affect the raw Lucene score), we call one of the search methods of the IndexSearcher, passing in the Weight object created by IndexSearcher.createNormalizedWeight(Query), Filter and the number of results we want. This method returns a TopDocs object, which is an internal collection of search results. The IndexSearcher creates a TopScoreDocCollector and passes it along with the Weight, Filter to another expert search method (for more on the Collector mechanism, see IndexSearcher). The TopScoreDocCollector uses a PriorityQueue to collect the top results for the search.

If a Filter is being used, some initial setup is done to determine which docs to include. Otherwise, we ask the Weight for a Scorer for each IndexReader segment and proceed by calling Scorer.score().

At last, we are actually going to score some documents. The score method takes in the Collector (most likely the TopScoreDocCollector or TopFieldCollector) and does its business.Of course, here is where things get involved. The Scorer that is returned by the Weight object depends on what type of Query was submitted. In most real world applications with multiple query terms, the Scorer is going to be a BooleanScorer2 created from BooleanWeight (see the section on custom queries for info on changing this).

Assuming a BooleanScorer2, we first initialize the Coordinator, which is used to apply the coord() factor. We then get a internal Scorer based on the required, optional and prohibited parts of the query. Using this internal Scorer, the BooleanScorer2 then proceeds into a while loop based on the Scorer.nextDoc() method. The nextDoc() method advances to the next document matching the query. This is an abstract method in the Scorer class and is thus overridden by all derived implementations. If you have a simple OR query your internal Scorer is most likely a DisjunctionSumScorer, which essentially combines the scorers from the sub scorers of the OR'd terms.

Copyright © 2000-2014 Apache Software Foundation. All Rights Reserved.