Package org.apache.lucene.index

package org.apache.lucene.index
Code to maintain and access indices.

Table Of Contents

  1. Index APIs
  2. Field types
  3. Postings APIs
  4. Index Statistics

Index APIs

IndexWriter

IndexWriter is used to create an index, and to add, update and delete documents. The IndexWriter class is thread safe, and enforces a single instance per index. Creating an IndexWriter creates a new index or opens an existing index for writing, in a Directory, depending on the configuration in IndexWriterConfig. A Directory is an abstraction that typically represents a local file-system directory (see various implementations of FSDirectory), but it may also stand for some other storage, such as RAM.

IndexReader

IndexReader is used to read data from the index, and supports searching. Many thread-safe readers may be open concurrently with a single (or no) writer. Each reader maintains a consistent "point in time" view of an index and must be explicitly refreshed (see DirectoryReader.openIfChanged(DirectoryReader, IndexWriter)) in order to incorporate writes that may occur after it is opened.

Segments and docids

Lucene's index is composed of segments, each of which contains a subset of all the documents in the index, and is a complete searchable index in itself, over that subset. As documents are written to the index, new segments are created and flushed to directory storage. Segments are composed of an immutable core and per-commit live documents and doc-value updates. Insertions add new segments. Deletions and doc-value updates in a given segment create a new segment that shares the same core as the previous segment and new live docs for this segment. Updates are implemented as an atomic insertion and deletion.

Over time, the writer merges groups of smaller segments into single larger ones in order to maintain an index that is efficient to search, and to reclaim dead space left behind by deleted (and updated) documents.

Each document is identified by a 32-bit number, its "docid," and is composed of a collection of Field values of diverse types (postings, stored fields, term vectors, doc values, points and knn vectors). Docids come in two flavors: global and per-segment. A document's global docid is just the sum of its per-segment docid and that segment's base docid offset. External, high-level APIs only handle global docids, but internal APIs that reference a LeafReader, which is a reader for a single segment, deal in per-segment docids.

Docids are assigned sequentially within each segment (starting at 0). Thus the number of documents in a segment is the same as its maximum docid; some may be deleted, but their docids are retained until the segment is merged. When segments merge, their documents are assigned new sequential docids. Accordingly, docid values must always be treated as internal implementation, not exposed as part of an application, nor stored or referenced outside of Lucene's internal APIs.

Field Types

Lucene supports a variety of different document field data structures. Lucene's core, the inverted index, is comprised of "postings." The postings, with their term dictionary, can be thought of as a map that provides efficient lookup given a Term (roughly, a word or token), to (the ordered list of) Documents containing that Term. Codecs may additionally record impacts alongside postings in order to be able to skip over low-scoring documents at search time. Postings do not provide any way of retrieving terms given a document, short of scanning the entire index.

Stored fields are essentially the opposite of postings, providing efficient retrieval of field values given a docid. All stored field values for a document are stored together in a block. Different types of stored field provide high-level datatypes such as strings and numbers on top of the underlying bytes. Stored field values are usually retrieved by the searcher using an implementation of StoredFieldVisitor.

TermVectors store a per-document inverted index. They are useful for finding similar documents, called MoreLikeThis in Lucene.

DocValues fields are what are sometimes referred to as columnar, or column-stride fields, by analogy to relational database terminology, in which documents are considered as rows, and fields, columns. DocValues fields store values per-field: a value for every document is held in a single data structure, providing for rapid, sequential lookup of a field-value given a docid. These fields are used for efficient value-based sorting, for faceting, and sometimes for filtering on the least selective clauses of a query.

PointValues represent numeric values using a kd-tree data structure. Efficient 1- and higher dimensional implementations make these the choice for numeric range and interval queries, and geo-spatial queries.

KnnVectorValues represent dense numeric vectors whose dimensions may either be bytes or floats. They are indexed in a way that allows searching for nearest neighbors. The vectors are typically produced by a machine-learned model, and used to perform semantic search.

Postings APIs

Terms

Terms represents the collection of terms within a field, exposes some metadata and statistics, and an API for enumeration. 

 Terms terms = leafReader.terms("body");
 // metadata about the field
 System.out.println("positions? " + terms.hasPositions());
 System.out.println("offsets? " + terms.hasOffsets());
 System.out.println("payloads? " + terms.hasPayloads());
 // iterate through terms
 TermsEnum termsEnum = terms.iterator();
 BytesRef term = null;
 while ((term = termsEnum.next()) != null) {
   doSomethingWith(term);
 }
TermsEnum provides an iterator over the list of terms within a field, some statistics about the term, and methods to access the term's documents and positions. 
 // seek to a specific term
 boolean found = termsEnum.seekExact(new BytesRef("foobar"));
 if (found) {
   // get the document frequency
   System.out.println(termsEnum.docFreq());
   // enumerate through documents
   PostingsEnum docs = termsEnum.postings(null);
   // enumerate through documents and positions
   PostingsEnum docsAndPositions = termsEnum.postings(null, PostingsEnum.POSITIONS);
 }

Documents

PostingsEnum is an extension of DocIdSetIterator that iterates over the list of documents for a term, along with the term frequency within that document. 

 int docid;
 while ((docid = docsEnum.nextDoc()) != DocIdSetIterator.NO_MORE_DOCS) {
   System.out.println(docid);
   System.out.println(docsEnum.freq());
 }

Positions

PostingsEnum also allows iteration of the positions a term occurred within the document, and any additional per-position information (offsets and payload). The information available is controlled by flags passed to TermsEnum#postings 

 int docid;
 PostingsEnum postings = termsEnum.postings(null, PostingsEnum.PAYLOADS | PostingsEnum.OFFSETS);
 while ((docid = postings.nextDoc()) != DocIdSetIterator.NO_MORE_DOCS) {
   System.out.println(docid);
   int freq = postings.freq();
   for (int i = 0; i < freq; i++) {
     System.out.println(postings.nextPosition());
     System.out.println(postings.startOffset());
     System.out.println(postings.endOffset());
     System.out.println(postings.getPayload());
   }
 }

Impacts

TermsEnum also allows returning an ImpactsEnum, an extension of PostingsEnum that exposes pareto-optimal tuples of (term frequency, length normalization factor) per block of postings. It is typically used to compute the maximum possible score over these blocks of postings, so that they can be skipped if they cannot possibly produce a competitive hit. 

 int docid;
 ImpactsEnum impactsEnum = termsEnum.impacts(PostingsEnum.FREQS);
 int targetDocID = 420;
 impactsEnum.advanceShallow(targetDocID);
 // These impacts expose pareto-optimal tuples of (termFreq, lengthNorm) over various ranges of doc IDs.
 Impacts impacts = impactsEnum.getImpacts();
 for (int level = 0; level < impacts.numLevels(); i++) {
   int docIdUpTo = impacts.getDocIdUpTo(level);
   // List of pareto-optimal (termFreq, lengthNorm) tuples between targetDocID inclusive and docIdUpTo inclusive.
   List<Impact> perLevelImpacts = impacts.getImpacts(level);
 }

Index Statistics

Term statistics

  • TermsEnum.docFreq(): Returns the number of documents that contain at least one occurrence of the term. This statistic is always available for an indexed term. Note that it will also count deleted documents, when segments are merged the statistic is updated as those deleted documents are merged away.
  • TermsEnum.totalTermFreq(): Returns the number of occurrences of this term across all documents. Like docFreq(), it will also count occurrences that appear in deleted documents.

Field statistics

  • Terms.size(): Returns the number of unique terms in the field. This statistic may be unavailable (returns -1) for some Terms implementations such as MultiTerms, where it cannot be efficiently computed. Note that this count also includes terms that appear only in deleted documents: when segments are merged such terms are also merged away and the statistic is then updated.
  • Terms.getDocCount(): Returns the number of documents that contain at least one occurrence of any term for this field. This can be thought of as a Field-level docFreq(). Like docFreq() it will also count deleted documents.
  • Terms.getSumDocFreq(): Returns the number of postings (term-document mappings in the inverted index) for the field. This can be thought of as the sum of TermsEnum.docFreq() across all terms in the field, and like docFreq() it will also count postings that appear in deleted documents.
  • Terms.getSumTotalTermFreq(): Returns the number of tokens for the field. This can be thought of as the sum of TermsEnum.totalTermFreq() across all terms in the field, and like totalTermFreq() it will also count occurrences that appear in deleted documents.

Segment statistics

Document statistics

Document statistics are available during the indexing process for an indexed field: typically a Similarity implementation will store some of these values (possibly in a lossy way), into the normalization value for the document in its Similarity.computeNorm(org.apache.lucene.index.FieldInvertState) method.

Additional user-supplied statistics can be added to the document as DocValues fields and accessed via LeafReader.getNumericDocValues(java.lang.String).