Package org.apache.lucene.util.hnsw
Class IncrementalHnswGraphMerger
java.lang.Object
org.apache.lucene.util.hnsw.IncrementalHnswGraphMerger
- All Implemented Interfaces:
HnswGraphMerger
- Direct Known Subclasses:
ConcurrentHnswMerger
This selects the biggest Hnsw graph from the provided merge state and initializes a new
HnswGraphBuilder with that graph as a starting point.
- WARNING: This API is experimental and might change in incompatible ways in the next release.
-
Field Summary
Modifier and TypeFieldDescriptionprotected final int
protected final FieldInfo
protected MergeState.DocMap
protected int
protected KnnVectorsReader
protected final int
protected final RandomVectorScorerSupplier
-
Constructor Summary
ConstructorDescriptionIncrementalHnswGraphMerger
(FieldInfo fieldInfo, RandomVectorScorerSupplier scorerSupplier, int M, int beamWidth) -
Method Summary
Modifier and TypeMethodDescriptionaddReader
(KnnVectorsReader reader, MergeState.DocMap docMap, Bits liveDocs) Adds a reader to the graph merger if it meets the following criteria: 1.protected HnswBuilder
createBuilder
(DocIdSetIterator mergedVectorIterator, int maxOrd) Builds a new HnswGraphBuilder using the biggest graph from the merge state as a starting point.protected final int[]
getNewOrdMapping
(DocIdSetIterator mergedVectorIterator, BitSet initializedNodes) Creates a new mapping from old ordinals to new ordinals and returns the total number of vectors in the newly merged segment.merge
(DocIdSetIterator mergedVectorIterator, InfoStream infoStream, int maxOrd) Merge and produce the on heap graph
-
Field Details
-
fieldInfo
-
scorerSupplier
-
M
protected final int M -
beamWidth
protected final int beamWidth -
initReader
-
initDocMap
-
initGraphSize
protected int initGraphSize
-
-
Constructor Details
-
IncrementalHnswGraphMerger
public IncrementalHnswGraphMerger(FieldInfo fieldInfo, RandomVectorScorerSupplier scorerSupplier, int M, int beamWidth) - Parameters:
fieldInfo
- FieldInfo for the field being merged
-
-
Method Details
-
addReader
public IncrementalHnswGraphMerger addReader(KnnVectorsReader reader, MergeState.DocMap docMap, Bits liveDocs) throws IOException Adds a reader to the graph merger if it meets the following criteria: 1. Does not contain any deleted docs 2. Is a HnswGraphProvider/PerFieldKnnVectorReader 3. Has the most docs of any previous reader that met the above criteria- Specified by:
addReader
in interfaceHnswGraphMerger
- Parameters:
reader
- KnnVectorsReader to add to the mergerdocMap
- MergeState.DocMap for the readerliveDocs
- Bits representing live docs, can be null- Returns:
- this
- Throws:
IOException
- If an error occurs while reading from the merge state
-
createBuilder
protected HnswBuilder createBuilder(DocIdSetIterator mergedVectorIterator, int maxOrd) throws IOException Builds a new HnswGraphBuilder using the biggest graph from the merge state as a starting point. If no valid readers were added to the merge state, a new graph is created.- Parameters:
mergedVectorIterator
- iterator over the vectors in the merged segmentmaxOrd
- max num of vectors that will be merged into the graph- Returns:
- HnswGraphBuilder
- Throws:
IOException
- If an error occurs while reading from the merge state
-
merge
public OnHeapHnswGraph merge(DocIdSetIterator mergedVectorIterator, InfoStream infoStream, int maxOrd) throws IOException Description copied from interface:HnswGraphMerger
Merge and produce the on heap graph- Specified by:
merge
in interfaceHnswGraphMerger
- Parameters:
mergedVectorIterator
- iterator over the vectors in the merged segmentinfoStream
- optional info stream to set to buildermaxOrd
- max number of vectors that will be added to the graph- Returns:
- merged graph
- Throws:
IOException
- during merge
-
getNewOrdMapping
protected final int[] getNewOrdMapping(DocIdSetIterator mergedVectorIterator, BitSet initializedNodes) throws IOException Creates a new mapping from old ordinals to new ordinals and returns the total number of vectors in the newly merged segment.- Parameters:
mergedVectorIterator
- iterator over the vectors in the merged segmentinitializedNodes
- track what nodes have been initialized- Returns:
- the mapping from old ordinals to new ordinals
- Throws:
IOException
- If an error occurs while reading from the merge state
-