Package org.apache.lucene.codecs.lucene99

Class Lucene99ScalarQuantizedVectorsFormat

java.lang.Object

org.apache.lucene.codecs.KnnVectorsFormat

org.apache.lucene.codecs.hnsw.FlatVectorsFormat

org.apache.lucene.codecs.lucene99.Lucene99ScalarQuantizedVectorsFormat

All Implemented Interfaces:

NamedSPILoader.NamedSPI

public class Lucene99ScalarQuantizedVectorsFormat
extends FlatVectorsFormat

Format supporting vector quantization, storage, and retrieval

WARNING: This API is experimental and might change in incompatible ways in the next release.

Field Summary

Fields

Modifier and Type	Field	Description
static final float	DYNAMIC_CONFIDENCE_INTERVAL	Dynamic confidence interval
static final String	NAME
static final String	QUANTIZED_VECTOR_COMPONENT

Fields inherited from class org.apache.lucene.codecs.KnnVectorsFormat

DEFAULT_MAX_DIMENSIONS, EMPTY

Constructor Summary

Constructors

Constructor	Description
Lucene99ScalarQuantizedVectorsFormat()	Constructs a format using default graph construction parameters
Lucene99ScalarQuantizedVectorsFormat(Float confidenceInterval, int bits, boolean compress)	Constructs a format using the given graph construction parameters.

Method Summary

Modifier and Type	Method	Description
static float	calculateDefaultConfidenceInterval(int vectorDimension)
FlatVectorsReader	fieldsReader(SegmentReadState state)	Returns a KnnVectorsReader to read the vectors from the index.
FlatVectorsWriter	fieldsWriter(SegmentWriteState state)	Returns a FlatVectorsWriter to write the vectors to the index.
String	toString()

Methods inherited from class org.apache.lucene.codecs.hnsw.FlatVectorsFormat

getMaxDimensions

Methods inherited from class org.apache.lucene.codecs.KnnVectorsFormat

availableKnnVectorsFormats, forName, getName, reloadKnnVectorsFormat

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

QUANTIZED_VECTOR_COMPONENT

public static final String QUANTIZED_VECTOR_COMPONENT

See Also:

• Constant Field Values

NAME

public static final String NAME

See Also:

• Constant Field Values

DYNAMIC_CONFIDENCE_INTERVAL

public static final float DYNAMIC_CONFIDENCE_INTERVAL

Dynamic confidence interval

See Also:

• Constant Field Values

Constructor Details

Lucene99ScalarQuantizedVectorsFormat

public Lucene99ScalarQuantizedVectorsFormat()

Constructs a format using default graph construction parameters

Lucene99ScalarQuantizedVectorsFormat

public Lucene99ScalarQuantizedVectorsFormat(Float confidenceInterval,
 int bits,
 boolean compress)

Constructs a format using the given graph construction parameters.

Parameters:

confidenceInterval - the confidenceInterval for scalar quantizing the vectors, when `null` it is calculated based on the vector dimension. When `0`, the quantiles are dynamically determined by sampling many confidence intervals and determining the most accurate pair.

bits - the number of bits to use for scalar quantization (must be between 1 and 8, inclusive)

compress - whether to compress the quantized vectors by another 50% when bits=4. If `true`, pairs of (4 bit quantized) dimensions are packed into a single byte. This must be `false` when bits=7. This provides a trade-off of 50% reduction in hot vector memory usage during searching, at some decode speed penalty.

Method Details

calculateDefaultConfidenceInterval

public static float calculateDefaultConfidenceInterval(int vectorDimension)

toString

public String toString()

Overrides:

toString in class Object

fieldsWriter

public FlatVectorsWriter fieldsWriter(SegmentWriteState state)
                               throws IOException

Description copied from class: FlatVectorsFormat

Returns a FlatVectorsWriter to write the vectors to the index.

Specified by:

fieldsWriter in class FlatVectorsFormat

Throws:

IOException

fieldsReader

public FlatVectorsReader fieldsReader(SegmentReadState state)
                               throws IOException

Description copied from class: FlatVectorsFormat

Returns a KnnVectorsReader to read the vectors from the index.

Specified by:

fieldsReader in class FlatVectorsFormat

Throws:

IOException