Package org.apache.lucene.index

Code to maintain and access indices.

See: Description

Interface Summary

Interface	Description
IndexableField	Represents a single field for indexing.
IndexableFieldType	Describes the properties of a field.
IndexReader.ReaderClosedListener	A custom listener that's invoked when the IndexReader is closed.
LeafReader.CoreClosedListener	Called when the shared core for this LeafReader is closed.
PointValues.IntersectVisitor	We recurse the BKD tree, using a provided instance of this to guide the recursion.
QueryTimeout	Base for query timeout implementations, which will provide a shouldExit() method, used with ExitableDirectoryReader.
TwoPhaseCommit	An interface for implementations that support 2-phase commit.

Class Summary

Class	Description
AutomatonTermsEnum	A FilteredTermsEnum that enumerates terms based upon what is accepted by a DFA.
BaseCompositeReader<R extends IndexReader>	Base class for implementing CompositeReaders based on an array of sub-readers.
BinaryDocValues	A per-document byte[]
CheckIndex	Basic tool and API to check the health of an index and write a new segments file that removes reference to problematic segments.
CheckIndex.Options	Run-time configuration options for CheckIndex commands.
CheckIndex.Status	Returned from CheckIndex.checkIndex() detailing the health and status of the index.
CheckIndex.Status.DocValuesStatus	Status from testing DocValues
CheckIndex.Status.FieldInfoStatus	Status from testing field infos.
CheckIndex.Status.FieldNormStatus	Status from testing field norms.
CheckIndex.Status.IndexSortStatus	Status from testing index sort
CheckIndex.Status.LiveDocStatus	Status from testing livedocs
CheckIndex.Status.PointsStatus	Status from testing PointValues
CheckIndex.Status.SegmentInfoStatus	Holds the status of each segment in the index.
CheckIndex.Status.StoredFieldStatus	Status from testing stored fields.
CheckIndex.Status.TermIndexStatus	Status from testing term index.
CheckIndex.Status.TermVectorStatus	Status from testing stored fields.
CheckIndex.VerifyPointsVisitor	Walks the entire N-dimensional points space, verifying that all points fall within the last cell's boundaries.
CodecReader	LeafReader implemented by codec APIs.
CompositeReader	Instances of this reader type can only be used to get stored fields from the underlying LeafReaders, but it is not possible to directly retrieve postings.
CompositeReaderContext	IndexReaderContext for CompositeReader instance.
ConcurrentMergeScheduler	A MergeScheduler that runs each merge using a separate thread.
DirectoryReader	DirectoryReader is an implementation of CompositeReader that can read indexes in a Directory.
DocIDMerger<T extends DocIDMerger.Sub>	Utility class to help merging documents from sub-readers according to either simple concatenated (unsorted) order, or by a specified index-time sort, skipping deleted documents and remapping non-deleted documents.
DocIDMerger.Sub	Represents one sub-reader being merged
DocValues	This class contains utility methods and constants for DocValues
ExitableDirectoryReader	The ExitableDirectoryReader wraps a real index DirectoryReader and allows for a QueryTimeout implementation object to be checked periodically to see if the thread should exit or not.
ExitableDirectoryReader.ExitableFields	Wrapper class for another Fields implementation that is used by the ExitableFilterAtomicReader.
ExitableDirectoryReader.ExitableFilterAtomicReader	Wrapper class for another FilterAtomicReader.
ExitableDirectoryReader.ExitableSubReaderWrapper	Wrapper class for a SubReaderWrapper that is used by the ExitableDirectoryReader.
ExitableDirectoryReader.ExitableTerms	Wrapper class for another Terms implementation that is used by ExitableFields.
ExitableDirectoryReader.ExitableTermsEnum	Wrapper class for TermsEnum that is used by ExitableTerms for implementing an exitable enumeration of terms.
FieldInfo	Access to the Field Info file that describes document fields and whether or not they are indexed.
FieldInfos	Collection of FieldInfos (accessible by number or by name).
FieldInvertState	This class tracks the number and position / offset parameters of terms being added to the index.
Fields	Flex API for access to fields and terms
FilterCodecReader	A FilterCodecReader contains another CodecReader, which it uses as its basic source of data, possibly transforming the data along the way or providing additional functionality.
FilterDirectoryReader	A FilterDirectoryReader wraps another DirectoryReader, allowing implementations to transform or extend it.
FilterDirectoryReader.SubReaderWrapper	Factory class passed to FilterDirectoryReader constructor that allows subclasses to wrap the filtered DirectoryReader's subreaders.
FilteredTermsEnum	Abstract class for enumerating a subset of all terms.
FilterLeafReader	A FilterLeafReader contains another LeafReader, which it uses as its basic source of data, possibly transforming the data along the way or providing additional functionality.
FilterLeafReader.FilterFields	Base class for filtering Fields implementations.
FilterLeafReader.FilterPostingsEnum	Base class for filtering PostingsEnum implementations.
FilterLeafReader.FilterTerms	Base class for filtering Terms implementations.
FilterLeafReader.FilterTermsEnum	Base class for filtering TermsEnum implementations.
IndexCommit	Expert: represents a single commit into an index as seen by the IndexDeletionPolicy or IndexReader.
IndexDeletionPolicy	Expert: policy for deletion of stale index commits.
IndexFileNames	This class contains useful constants representing filenames and extensions used by lucene, as well as convenience methods for querying whether a file name matches an extension (matchesExtension), as well as generating file names from a segment name, generation and extension ( fileNameFromGeneration, segmentFileName).
IndexReader	IndexReader is an abstract class, providing an interface for accessing a point-in-time view of an index.
IndexReaderContext	A struct like class that represents a hierarchical relationship between IndexReader instances.
IndexUpgrader	This is an easy-to-use tool that upgrades all segments of an index from previous Lucene versions to the current segment file format.
IndexWriter	An IndexWriter creates and maintains an index.
IndexWriter.IndexReaderWarmer	If DirectoryReader.open(IndexWriter) has been called (ie, this writer is in near real-time mode), then after a merge completes, this class can be invoked to warm the reader on the newly merged segment, before the merge commits.
IndexWriterConfig	Holds all the configuration that is used to create an IndexWriter.
KeepOnlyLastCommitDeletionPolicy	This IndexDeletionPolicy implementation that keeps only the most recent commit and immediately removes all prior commits after a new commit is done.
LeafReader	LeafReader is an abstract class, providing an interface for accessing an index.
LeafReaderContext	IndexReaderContext for LeafReader instances.
LiveIndexWriterConfig	Holds all the configuration used by IndexWriter with few setters for settings that can be changed on an IndexWriter instance "live".
LogByteSizeMergePolicy	This is a LogMergePolicy that measures size of a segment as the total byte size of the segment's files.
LogDocMergePolicy	This is a LogMergePolicy that measures size of a segment as the number of documents (not taking deletions into account).
LogMergePolicy	This class implements a MergePolicy that tries to merge segments into levels of exponentially increasing size, where each level has fewer segments than the value of the merge factor.
MappedMultiFields	A Fields implementation that merges multiple Fields into one, and maps around deleted documents.
MergePolicy	Expert: a MergePolicy determines the sequence of primitive merge operations.
MergePolicy.MergeSpecification	A MergeSpecification instance provides the information necessary to perform multiple merges.
MergePolicy.OneMerge	OneMerge provides the information necessary to perform an individual primitive merge operation, resulting in a single new segment.
MergePolicy.OneMergeProgress	Progress and state for an executing merge.
MergePolicyWrapper	A wrapper for MergePolicy instances.
MergeRateLimiter	This is the RateLimiter that IndexWriter assigns to each running merge, to give MergeSchedulers ionice like control.
MergeScheduler	Expert: IndexWriter uses an instance implementing this interface to execute the merges selected by a MergePolicy.
MergeState	Holds common state used during segment merging.
MergeState.DocMap	A map of doc IDs.
MultiDocValues	A wrapper for CompositeIndexReader providing access to DocValues.
MultiDocValues.MultiSortedDocValues	Implements SortedDocValues over n subs, using an OrdinalMap
MultiDocValues.MultiSortedSetDocValues	Implements MultiSortedSetDocValues over n subs, using an OrdinalMap
MultiDocValues.OrdinalMap	maps per-segment ordinals to/from global ordinal space
MultiFields	Exposes flex API, merged from flex API of sub-segments.
MultiPostingsEnum	Exposes PostingsEnum, merged from PostingsEnum API of sub-segments.
MultiPostingsEnum.EnumWithSlice	Holds a PostingsEnum along with the corresponding ReaderSlice.
MultiReader	A CompositeReader which reads multiple indexes, appending their content.
MultiTerms	Exposes flex API, merged from flex API of sub-segments.
MultiTermsEnum	Exposes TermsEnum API, merged from TermsEnum API of sub-segments.
NoDeletionPolicy	An IndexDeletionPolicy which keeps all index commits around, never deleting them.
NoMergePolicy	A MergePolicy which never returns merges to execute.
NoMergeScheduler	A MergeScheduler which never executes any merges.
NumericDocValues	A per-document numeric value.
OneMergeWrappingMergePolicy	A wrapping merge policy that wraps the MergePolicy.OneMerge objects returned by the wrapped merge policy.
OrdTermState	An ordinal based TermState
ParallelCompositeReader	An CompositeReader which reads multiple, parallel indexes.
ParallelLeafReader	An LeafReader which reads multiple, parallel indexes.
PersistentSnapshotDeletionPolicy	A SnapshotDeletionPolicy which adds a persistence layer so that snapshots can be maintained across the life of an application.
PointValues	Access to indexed numeric values.
PostingsEnum	Iterates through the postings.
PrefixCodedTerms	Prefix codes term instances (prefixes are shared).
PrefixCodedTerms.Builder	Builds a PrefixCodedTerms: call add repeatedly, then finish.
PrefixCodedTerms.TermIterator	An iterator over the list of terms stored in a PrefixCodedTerms.
QueryTimeoutImpl	An implementation of QueryTimeout that can be used by the ExitableDirectoryReader class to time out and exit out when a query takes a long time to rewrite.
RandomAccessOrds	Extension of SortedSetDocValues that supports random access to the ordinals of a document.
ReaderManager	Utility class to safely share DirectoryReader instances across multiple threads, while periodically reopening.
ReaderSlice	Subreader slice from a parent composite reader.
ReaderUtil	Common util methods for dealing with IndexReaders and IndexReaderContexts.
SegmentCommitInfo	Embeds a [read-only] SegmentInfo and adds per-commit fields.
SegmentInfo	Information about a segment such as its name, directory, and files related to the segment.
SegmentInfos	A collection of segmentInfo objects with methods for operating on those segments in relation to the file system.
SegmentInfos.FindSegmentsFile<T>	Utility class for executing code that needs to do something with the current segments file.
SegmentReader	IndexReader implementation over a single segment.
SegmentReadState	Holder class for common parameters used during read.
SegmentWriteState	Holder class for common parameters used during write.
SerialMergeScheduler	A MergeScheduler that simply does each merge sequentially, using the current thread.
SimpleMergedSegmentWarmer	A very simple merged segment warmer that just ensures data structures are initialized.
SingleTermsEnum	Subclass of FilteredTermsEnum for enumerating a single term.
SlowCodecReaderWrapper	Wraps arbitrary readers for merging.
SnapshotDeletionPolicy	An IndexDeletionPolicy that wraps any other IndexDeletionPolicy and adds the ability to hold and later release snapshots of an index.
SortedDocValues	A per-document byte[] with presorted values.
SortedNumericDocValues	A list of per-document numeric values, sorted according to Long.compare(long, long).
SortedSetDocValues	A per-document set of presorted byte[] values.
StandardDirectoryReader	Default implementation of DirectoryReader.
StoredFieldVisitor	Expert: provides a low-level means of accessing the stored field values in an index.
Term	A Term represents a word from text.
TermContext	Maintains a IndexReader TermState view over IndexReader instances containing a single term.
Terms	Access to the terms in a specific field.
TermsEnum	Iterator to seek (TermsEnum.seekCeil(BytesRef), TermsEnum.seekExact(BytesRef)) or step through (BytesRefIterator.next() terms to obtain frequency information (TermsEnum.docFreq()), PostingsEnum or PostingsEnum for the current term (TermsEnum.postings(org.apache.lucene.index.PostingsEnum).
TermState	Encapsulates all required internal state to position the associated TermsEnum without re-seeking.
TieredMergePolicy	Merges segments of approximately equal size, subject to an allowed number of segments per tier.
TieredMergePolicy.MergeScore	Holds score and explanation for a single candidate merge.
TwoPhaseCommitTool	A utility for executing 2-phase commit on several objects.
UpgradeIndexMergePolicy	This MergePolicy is used for upgrading all existing segments of an index when calling IndexWriter.forceMerge(int).

Enum Summary

Enum	Description
DocValuesType	DocValues types.
FilteredTermsEnum.AcceptStatus	Return value, if term should be accepted or the iteration should END.
IndexOptions	Controls how much information is stored in the postings lists.
IndexWriterConfig.OpenMode	Specifies the open mode for IndexWriter.
MergePolicy.OneMergeProgress.PauseReason	Reason for pausing the merge thread.
MergeTrigger	MergeTrigger is passed to MergePolicy.findMerges(MergeTrigger, org.apache.lucene.index.SegmentInfos, IndexWriter) to indicate the event that triggered the merge.
PointValues.Relation	Used by PointValues.intersect(java.lang.String, org.apache.lucene.index.PointValues.IntersectVisitor) to check how each recursive cell corresponds to the query.
StoredFieldVisitor.Status	Enumeration of possible return values for StoredFieldVisitor.needsField(org.apache.lucene.index.FieldInfo).
TermsEnum.SeekStatus	Represents returned result from TermsEnum.seekCeil(org.apache.lucene.util.BytesRef).

Exception Summary

Exception	Description
CorruptIndexException	This exception is thrown when Lucene detects an inconsistency in the index.
ExitableDirectoryReader.ExitingReaderException	Exception that is thrown to prematurely terminate a term enumeration.
IndexFormatTooNewException	This exception is thrown when Lucene detects an index that is newer than this Lucene version.
IndexFormatTooOldException	This exception is thrown when Lucene detects an index that is too old for this Lucene version
IndexNotFoundException	Signals that no index was found in the Directory.
MergePolicy.MergeAbortedException	Thrown when a merge was explicitly aborted because IndexWriter.abortMerges() was called.
MergePolicy.MergeException	Exception thrown if there are any problems while executing a merge.
TwoPhaseCommitTool.CommitFailException	Thrown by TwoPhaseCommitTool.execute(TwoPhaseCommit...) when an object fails to commit().
TwoPhaseCommitTool.PrepareCommitFailException	Thrown by TwoPhaseCommitTool.execute(TwoPhaseCommit...) when an object fails to prepareCommit().

Package org.apache.lucene.index Description

Code to maintain and access indices.

Table Of Contents

1. Postings APIs
   • Fields
   • Terms
   • Documents
   • Positions
2. Index Statistics
   • Term-level
   • Field-level
   • Segment-level
   • Document-level

Postings APIs

Fields

Fields is the initial entry point into the postings APIs, this can be obtained in several ways:

 // access indexed fields for an index segment
 Fields fields = reader.fields();
 // access term vector fields for a specified document
 Fields fields = reader.getTermVectors(docid);
 

Fields implements Java's Iterable interface, so it's easy to enumerate the list of fields:

 // enumerate list of fields
 for (String field : fields) {
   // access the terms for this field
   Terms terms = fields.terms(field);
 }
 

Terms

Terms represents the collection of terms within a field, exposes some metadata and statistics, and an API for enumeration.

 // metadata about the field
 System.out.println("positions? " + terms.hasPositions());
 System.out.println("offsets? " + terms.hasOffsets());
 System.out.println("payloads? " + terms.hasPayloads());
 // iterate through terms
 TermsEnum termsEnum = terms.iterator(null);
 BytesRef term = null;
 while ((term = termsEnum.next()) != null) {
   doSomethingWith(termsEnum.term());
 }
 

TermsEnum provides an iterator over the list of terms within a field, some statistics about the term, and methods to access the term's documents and positions.

 // seek to a specific term
 boolean found = termsEnum.seekExact(new BytesRef("foobar"));
 if (found) {
   // get the document frequency
   System.out.println(termsEnum.docFreq());
   // enumerate through documents
   PostingsEnum docs = termsEnum.postings(null, null);
   // enumerate through documents and positions
   PostingsEnum docsAndPositions = termsEnum.postings(null, null, PostingsEnum.FLAG_POSITIONS);
 }
 

Documents

PostingsEnum is an extension of DocIdSetIteratorthat iterates over the list of documents for a term, along with the term frequency within that document.

 int docid;
 while ((docid = docsEnum.nextDoc()) != DocIdSetIterator.NO_MORE_DOCS) {
   System.out.println(docid);
   System.out.println(docsEnum.freq());
  }
 

Positions

PostingsEnum also allows iteration of the positions a term occurred within the document, and any additional per-position information (offsets and payload). The information available is controlled by flags passed to TermsEnum#postings

 int docid;
 PostingsEnum postings = termsEnum.postings(null, null, PostingsEnum.FLAG_PAYLOADS | PostingsEnum.FLAG_OFFSETS);
 while ((docid = postings.nextDoc()) != DocIdSetIterator.NO_MORE_DOCS) {
   System.out.println(docid);
   int freq = postings.freq();
   for (int i = 0; i < freq; i++) {
      System.out.println(postings.nextPosition());
      System.out.println(postings.startOffset());
      System.out.println(postings.endOffset());
      System.out.println(postings.getPayload());
   }
 }
 

Index Statistics

Term statistics

• TermsEnum.docFreq(): Returns the number of documents that contain at least one occurrence of the term. This statistic is always available for an indexed term. Note that it will also count deleted documents, when segments are merged the statistic is updated as those deleted documents are merged away.
• TermsEnum.totalTermFreq(): Returns the number of occurrences of this term across all documents. Note that this statistic is unavailable (returns -1) if term frequencies were omitted from the index (DOCS) for the field. Like docFreq(), it will also count occurrences that appear in deleted documents.

Field statistics

• Terms.size(): Returns the number of unique terms in the field. This statistic may be unavailable (returns -1) for some Terms implementations such as MultiTerms, where it cannot be efficiently computed. Note that this count also includes terms that appear only in deleted documents: when segments are merged such terms are also merged away and the statistic is then updated.
• Terms.getDocCount(): Returns the number of documents that contain at least one occurrence of any term for this field. This can be thought of as a Field-level docFreq(). Like docFreq() it will also count deleted documents.
• Terms.getSumDocFreq(): Returns the number of postings (term-document mappings in the inverted index) for the field. This can be thought of as the sum of TermsEnum.docFreq() across all terms in the field, and like docFreq() it will also count postings that appear in deleted documents.
• Terms.getSumTotalTermFreq(): Returns the number of tokens for the field. This can be thought of as the sum of TermsEnum.totalTermFreq() across all terms in the field, and like totalTermFreq() it will also count occurrences that appear in deleted documents, and will be unavailable (returns -1) if term frequencies were omitted from the index (DOCS) for the field.

Segment statistics

• IndexReader.maxDoc(): Returns the number of documents (including deleted documents) in the index.
• IndexReader.numDocs(): Returns the number of live documents (excluding deleted documents) in the index.
• IndexReader.numDeletedDocs(): Returns the number of deleted documents in the index.
• Fields.size(): Returns the number of indexed fields.

Document statistics

Document statistics are available during the indexing process for an indexed field: typically a Similarity implementation will store some of these values (possibly in a lossy way), into the normalization value for the document in its Similarity.computeNorm(org.apache.lucene.index.FieldInvertState) method.

• FieldInvertState.getLength(): Returns the number of tokens for this field in the document. Note that this is just the number of times that TokenStream.incrementToken() returned true, and is unrelated to the values in PositionIncrementAttribute.
• FieldInvertState.getNumOverlap(): Returns the number of tokens for this field in the document that had a position increment of zero. This can be used to compute a document length that discounts artificial tokens such as synonyms.
• FieldInvertState.getPosition(): Returns the accumulated position value for this field in the document: computed from the values of PositionIncrementAttribute and including Analyzer.getPositionIncrementGap(java.lang.String)s across multivalued fields.
• FieldInvertState.getOffset(): Returns the total character offset value for this field in the document: computed from the values of OffsetAttribute returned by TokenStream.end(), and including Analyzer.getOffsetGap(java.lang.String)s across multivalued fields.
• FieldInvertState.getUniqueTermCount(): Returns the number of unique terms encountered for this field in the document.
• FieldInvertState.getMaxTermFrequency(): Returns the maximum frequency across all unique terms encountered for this field in the document.

Additional user-supplied statistics can be added to the document as DocValues fields and accessed via LeafReader.getNumericDocValues(java.lang.String).