Package org.apache.lucene.util.hnsw

Class IncrementalHnswGraphMerger

java.lang.Object

org.apache.lucene.util.hnsw.IncrementalHnswGraphMerger

All Implemented Interfaces:

HnswGraphMerger

Direct Known Subclasses:

ConcurrentHnswMerger

public class IncrementalHnswGraphMerger
extends Object
implements HnswGraphMerger

This selects the biggest Hnsw graph from the provided merge state and initializes a new HnswGraphBuilder with that graph as a starting point.

WARNING: This API is experimental and might change in incompatible ways in the next release.

Field Summary

Fields

Modifier and Type	Field	Description
protected int	beamWidth
protected FieldInfo	fieldInfo
protected MergeState.DocMap	initDocMap
protected int	initGraphSize
protected KnnVectorsReader	initReader
protected int	M
protected RandomVectorScorerSupplier	scorerSupplier

Constructor Summary

Constructors

Constructor	Description
IncrementalHnswGraphMerger(FieldInfo fieldInfo, RandomVectorScorerSupplier scorerSupplier, int M, int beamWidth)

Method Summary

Modifier and Type	Method	Description
IncrementalHnswGraphMerger	addReader(KnnVectorsReader reader, MergeState.DocMap docMap, Bits liveDocs)	Adds a reader to the graph merger if it meets the following criteria: 1.
protected HnswBuilder	createBuilder(DocIdSetIterator mergedVectorIterator, int maxOrd)	Builds a new HnswGraphBuilder using the biggest graph from the merge state as a starting point.
protected int[]	getNewOrdMapping(DocIdSetIterator mergedVectorIterator, BitSet initializedNodes)	Creates a new mapping from old ordinals to new ordinals and returns the total number of vectors in the newly merged segment.
OnHeapHnswGraph	merge(DocIdSetIterator mergedVectorIterator, InfoStream infoStream, int maxOrd)	Merge and produce the on heap graph

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

fieldInfo

protected final FieldInfo fieldInfo

scorerSupplier

protected final RandomVectorScorerSupplier scorerSupplier

M

protected final int M

beamWidth

protected final int beamWidth

initReader

protected KnnVectorsReader initReader

initDocMap

protected MergeState.DocMap initDocMap

initGraphSize

protected int initGraphSize

Constructor Detail

IncrementalHnswGraphMerger

public IncrementalHnswGraphMerger(FieldInfo fieldInfo,
                                  RandomVectorScorerSupplier scorerSupplier,
                                  int M,
                                  int beamWidth)

Parameters:

fieldInfo - FieldInfo for the field being merged

Method Detail

addReader

public IncrementalHnswGraphMerger addReader(KnnVectorsReader reader,
                                            MergeState.DocMap docMap,
                                            Bits liveDocs)
                                     throws IOException

Adds a reader to the graph merger if it meets the following criteria: 1. Does not contain any deleted docs 2. Is a HnswGraphProvider/PerFieldKnnVectorReader 3. Has the most docs of any previous reader that met the above criteria

Specified by:

addReader in interface HnswGraphMerger

Parameters:

reader - KnnVectorsReader to add to the merger

docMap - MergeState.DocMap for the reader

liveDocs - Bits representing live docs, can be null

Returns:

this

Throws:

IOException - If an error occurs while reading from the merge state

createBuilder

protected HnswBuilder createBuilder(DocIdSetIterator mergedVectorIterator,
                                    int maxOrd)
                             throws IOException

Builds a new HnswGraphBuilder using the biggest graph from the merge state as a starting point. If no valid readers were added to the merge state, a new graph is created.

Parameters:

mergedVectorIterator - iterator over the vectors in the merged segment

maxOrd - max num of vectors that will be merged into the graph

Returns:

HnswGraphBuilder

Throws:

IOException - If an error occurs while reading from the merge state

merge

public OnHeapHnswGraph merge(DocIdSetIterator mergedVectorIterator,
                             InfoStream infoStream,
                             int maxOrd)
                      throws IOException

Description copied from interface: HnswGraphMerger

Merge and produce the on heap graph

Specified by:

merge in interface HnswGraphMerger

Parameters:

mergedVectorIterator - iterator over the vectors in the merged segment

infoStream - optional info stream to set to builder

maxOrd - max number of vectors that will be added to the graph

Returns:

merged graph

Throws:

IOException - during merge

getNewOrdMapping

protected final int[] getNewOrdMapping(DocIdSetIterator mergedVectorIterator,
                                       BitSet initializedNodes)
                                throws IOException

Creates a new mapping from old ordinals to new ordinals and returns the total number of vectors in the newly merged segment.

Parameters:

mergedVectorIterator - iterator over the vectors in the merged segment

initializedNodes - track what nodes have been initialized

Returns:

the mapping from old ordinals to new ordinals

Throws:

IOException - If an error occurs while reading from the merge state