org.apache.lucene.codecs.lucene49

Class Lucene49DocValuesFormat

java.lang.Object

org.apache.lucene.codecs.DocValuesFormat

org.apache.lucene.codecs.lucene49.Lucene49DocValuesFormat

All Implemented Interfaces:

NamedSPILoader.NamedSPI

public class Lucene49DocValuesFormat
extends DocValuesFormat

Lucene 4.9 DocValues format.

Encodes the five per-document value types (Numeric,Binary,Sorted,SortedSet,SortedNumeric) with these strategies:

NUMERIC:

• Delta-compressed: per-document integers written as deltas from the minimum value, compressed with bitpacking. For more information, see DirectWriter.
• Table-compressed: when the number of unique values is very small (< 256), and when there are unused "gaps" in the range of values used (such as SmallFloat), a lookup table is written instead. Each per-document entry is instead the ordinal to this table, and those ordinals are compressed with bitpacking (DirectWriter).
• GCD-compressed: when all numbers share a common divisor, such as dates, the greatest common denominator (GCD) is computed, and quotients are stored using Delta-compressed Numerics.
• Monotonic-compressed: when all numbers are monotonically increasing offsets, they are written as blocks of bitpacked integers, encoding the deviation from the expected delta.

BINARY:

• Fixed-width Binary: one large concatenated byte[] is written, along with the fixed length. Each document's value can be addressed directly with multiplication (docID * length).
• Variable-width Binary: one large concatenated byte[] is written, along with end addresses for each document. The addresses are written as Monotonic-compressed numerics.
• Prefix-compressed Binary: values are written in chunks of 16, with the first value written completely and other values sharing prefixes. chunk addresses are written as Monotonic-compressed numerics.

SORTED:

• Sorted: a mapping of ordinals to deduplicated terms is written as Prefix-Compressed Binary, along with the per-document ordinals written using one of the numeric strategies above.

SORTED_SET:

• SortedSet: a mapping of ordinals to deduplicated terms is written as Prefix-Compressed Binary, an ordinal list and per-document index into this list are written using the numeric strategies above.

SORTED_NUMERIC:

• SortedNumeric: a value list and per-document index into this list are written using the numeric strategies above.

Files:

1. .dvd: DocValues data
2. .dvm: DocValues metadata

1. The DocValues metadata or .dvm file.

   For DocValues field, this stores metadata, such as the offset into the DocValues data (.dvd)

   DocValues metadata (.dvm) --> Header,<Entry>^NumFields,Footer

   • Entry --> NumericEntry | BinaryEntry | SortedEntry | SortedSetEntry | SortedNumericEntry
   • NumericEntry --> GCDNumericEntry | TableNumericEntry | DeltaNumericEntry
   • GCDNumericEntry --> NumericHeader,MinValue,GCD,BitsPerValue
   • TableNumericEntry --> NumericHeader,TableSize,Int64^TableSize,BitsPerValue
   • DeltaNumericEntry --> NumericHeader,MinValue,BitsPerValue
   • MonotonicNumericEntry --> NumericHeader,PackedVersion,BlockSize
   • NumericHeader --> FieldNumber,EntryType,NumericType,MissingOffset,DataOffset,Count,EndOffset
   • BinaryEntry --> FixedBinaryEntry | VariableBinaryEntry | PrefixBinaryEntry
   • FixedBinaryEntry --> BinaryHeader
   • VariableBinaryEntry --> BinaryHeader,AddressOffset,PackedVersion,BlockSize
   • PrefixBinaryEntry --> BinaryHeader,AddressInterval,AddressOffset,PackedVersion,BlockSize
   • BinaryHeader --> FieldNumber,EntryType,BinaryType,MissingOffset,MinLength,MaxLength,DataOffset
   • SortedEntry --> FieldNumber,EntryType,BinaryEntry,NumericEntry
   • SortedSetEntry --> EntryType,BinaryEntry,NumericEntry,NumericEntry
   • SortedNumericEntry --> EntryType,NumericEntry,NumericEntry
   • FieldNumber,PackedVersion,MinLength,MaxLength,BlockSize,ValueCount --> VInt
   • EntryType,CompressionType --> Byte
   • Header --> CodecHeader
   • MinValue,GCD,MissingOffset,AddressOffset,DataOffset,EndOffset --> Int64
   • TableSize,BitsPerValue --> vInt
   • Footer --> CodecFooter

   Sorted fields have two entries: a BinaryEntry with the value metadata, and an ordinary NumericEntry for the document-to-ord metadata.

   SortedSet fields have three entries: a BinaryEntry with the value metadata, and two NumericEntries for the document-to-ord-index and ordinal list metadata.

   SortedNumeric fields have two entries: A NumericEntry with the value metadata, and a numeric entry with the document-to-value index.

   FieldNumber of -1 indicates the end of metadata.

   EntryType is a 0 (NumericEntry) or 1 (BinaryEntry)

   DataOffset is the pointer to the start of the data in the DocValues data (.dvd)

   EndOffset is the pointer to the end of the data in the DocValues data (.dvd)

   NumericType indicates how Numeric values will be compressed:

   • 0 --> delta-compressed. For each block of 16k integers, every integer is delta-encoded from the minimum value within the block.
   • 1 -->, gcd-compressed. When all integers share a common divisor, only quotients are stored using blocks of delta-encoded ints.
   • 2 --> table-compressed. When the number of unique numeric values is small and it would save space, a lookup table of unique values is written, followed by the ordinal for each document.

   BinaryType indicates how Binary values will be stored:

   • 0 --> fixed-width. All values have the same length, addressing by multiplication.
   • 1 -->, variable-width. An address for each value is stored.
   • 2 --> prefix-compressed. An address to the start of every interval'th value is stored.

   MinLength and MaxLength represent the min and max byte[] value lengths for Binary values. If they are equal, then all values are of a fixed size, and can be addressed as DataOffset + (docID * length). Otherwise, the binary values are of variable size, and packed integer metadata (PackedVersion,BlockSize) is written for the addresses.

   MissingOffset points to a byte[] containing a bitset of all documents that had a value for the field. If its -1, then there are no missing values.

   Checksum contains the CRC32 checksum of all bytes in the .dvm file up until the checksum. This is used to verify integrity of the file on opening the index.

2. The DocValues data or .dvd file.

   For DocValues field, this stores the actual per-document data (the heavy-lifting)

   DocValues data (.dvd) --> Header,<NumericData | BinaryData | SortedData>^NumFields,Footer

   • NumericData --> DeltaCompressedNumerics | TableCompressedNumerics | GCDCompressedNumerics
   • BinaryData --> Byte^DataLength,Addresses
   • SortedData --> FST<Int64>
   • DeltaCompressedNumerics,TableCompressedNumerics,GCDCompressedNumerics --> PackedInts
   • Addresses --> MonotonicBlockPackedInts(blockSize=16k)
   • Footer --> CodecFooter

WARNING: This API is experimental and might change in incompatible ways in the next release.

Constructor Summary

Constructors

Constructor and Description
Lucene49DocValuesFormat() Sole Constructor

Method Summary

Methods

Modifier and Type	Method and Description
DocValuesConsumer	fieldsConsumer(SegmentWriteState state) Returns a DocValuesConsumer to write docvalues to the index.
DocValuesProducer	fieldsProducer(SegmentReadState state) Returns a DocValuesProducer to read docvalues from the index.

Methods inherited from class org.apache.lucene.codecs.DocValuesFormat

availableDocValuesFormats, forName, getName, reloadDocValuesFormats, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

Lucene49DocValuesFormat

public Lucene49DocValuesFormat()

Sole Constructor

Method Detail

fieldsConsumer

public DocValuesConsumer fieldsConsumer(SegmentWriteState state)
                                 throws IOException

Description copied from class: DocValuesFormat

Returns a DocValuesConsumer to write docvalues to the index.

Specified by:

fieldsConsumer in class DocValuesFormat

Throws:

IOException

fieldsProducer

public DocValuesProducer fieldsProducer(SegmentReadState state)
                                 throws IOException

Description copied from class: DocValuesFormat

Returns a DocValuesProducer to read docvalues from the index.

NOTE: by the time this call returns, it must hold open any files it will need to use; else, those files may be deleted. Additionally, required files may be deleted during the execution of this call before there is a chance to open them. Under these circumstances an IOException should be thrown by the implementation. IOExceptions are expected and will automatically cause a retry of the segment opening logic with the newly revised segments.

Specified by:

fieldsProducer in class DocValuesFormat

Throws:

IOException