org.apache.lucene.codecs.lucene49

Class Lucene49NormsFormat

java.lang.Object

org.apache.lucene.codecs.NormsFormat

org.apache.lucene.codecs.lucene49.Lucene49NormsFormat

public class Lucene49NormsFormat
extends NormsFormat

Lucene 4.9 Score normalization format.

Encodes normalization values with these strategies:

• Uncompressed: when values fit into a single byte and would require more than 4 bits per value, they are just encoded as an uncompressed byte array.
• Constant: when there is only one value present for the entire field, no actual data is written: this constant is encoded in the metadata
• Table-compressed: when the number of unique values is very small (< 64), and when there are unused "gaps" in the range of values used (such as SmallFloat), a lookup table is written instead. Each per-document entry is instead the ordinal to this table, and those ordinals are compressed with bitpacking (PackedInts).
• Delta-compressed: per-document integers written as deltas from the minimum value, compressed with bitpacking. For more information, see BlockPackedWriter. This is only used when norms of larger than one byte are present.

Files:

1. .nvd: Norms data
2. .nvm: Norms metadata

1. The Norms metadata or .nvm file.

   For each norms field, this stores metadata, such as the offset into the Norms data (.nvd)

   Norms metadata (.dvm) --> Header,<Entry>^NumFields,Footer

   • Header --> CodecHeader
   • Entry --> FieldNumber,Type,Offset
   • FieldNumber --> vInt
   • Type --> Byte
   • Offset --> Int64
   • Footer --> CodecFooter

   FieldNumber of -1 indicates the end of metadata.

   Offset is the pointer to the start of the data in the norms data (.nvd), or the singleton value for Constant

   Type indicates how Numeric values will be compressed:

   • 0 --> delta-compressed. For each block of 16k integers, every integer is delta-encoded from the minimum value within the block.
   • 1 --> table-compressed. When the number of unique numeric values is small and it would save space, a lookup table of unique values is written, followed by the ordinal for each document.
   • 2 --> constant. When there is a single value for the entire field.
   • 3 --> uncompressed: Values written as a simple byte[].

2. The Norms data or .nvd file.

   For each Norms field, this stores the actual per-document data (the heavy-lifting)

   Norms data (.nvd) --> Header,<Uncompressed | TableCompressed | DeltaCompressed>^NumFields,Footer

   • Header --> CodecHeader
   • Uncompressed --> Byte^maxDoc
   • TableCompressed --> PackedIntsVersion,Table,BitPackedData
   • Table --> TableSize, int64^TableSize
   • BitpackedData --> PackedInts
   • DeltaCompressed --> PackedIntsVersion,BlockSize,DeltaCompressedData
   • DeltaCompressedData --> BlockPackedWriter(blockSize=16k)
   • PackedIntsVersion,BlockSize,TableSize --> vInt
   • Footer --> CodecFooter

WARNING: This API is experimental and might change in incompatible ways in the next release.

Constructor Summary

Constructors

Constructor and Description
Lucene49NormsFormat() Sole Constructor

Method Summary

Methods

Modifier and Type	Method and Description
DocValuesConsumer	normsConsumer(SegmentWriteState state) Returns a DocValuesConsumer to write norms to the index.
DocValuesProducer	normsProducer(SegmentReadState state) Returns a DocValuesProducer to read norms from the index.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Lucene49NormsFormat

public Lucene49NormsFormat()

Sole Constructor

Method Detail

normsConsumer

public DocValuesConsumer normsConsumer(SegmentWriteState state)
                                throws IOException

Description copied from class: NormsFormat

Returns a DocValuesConsumer to write norms to the index.

Specified by:

normsConsumer in class NormsFormat

Throws:

IOException

normsProducer

public DocValuesProducer normsProducer(SegmentReadState state)
                                throws IOException

Description copied from class: NormsFormat

Returns a DocValuesProducer to read norms from the index.

NOTE: by the time this call returns, it must hold open any files it will need to use; else, those files may be deleted. Additionally, required files may be deleted during the execution of this call before there is a chance to open them. Under these circumstances an IOException should be thrown by the implementation. IOExceptions are expected and will automatically cause a retry of the segment opening logic with the newly revised segments.

Specified by:

normsProducer in class NormsFormat

Throws:

IOException