Package org.apache.lucene.tests.index

Class BaseKnnVectorsFormatTestCase

java.lang.Object

org.junit.Assert

org.apache.lucene.tests.util.LuceneTestCase

org.apache.lucene.tests.index.BaseIndexFileFormatTestCase

org.apache.lucene.tests.index.BaseKnnVectorsFormatTestCase

public abstract class BaseKnnVectorsFormatTestCase
extends BaseIndexFileFormatTestCase

Base class aiming at testing vectors formats. To test a new format, all you need is to register a new Codec which uses it and extend this class and override BaseIndexFileFormatTestCase.getCodec().

WARNING: This API is experimental and might change in incompatible ways in the next release.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.tests.index.BaseIndexFileFormatTestCase

BaseIndexFileFormatTestCase.FileTrackingDirectoryWrapper, BaseIndexFileFormatTestCase.ReadBytesDirectoryWrapper

Nested classes/interfaces inherited from class org.apache.lucene.tests.util.LuceneTestCase

LuceneTestCase.AwaitsFix, LuceneTestCase.BadApple, LuceneTestCase.Monster, LuceneTestCase.Nightly, LuceneTestCase.SuppressCodecs, LuceneTestCase.SuppressFileSystems, LuceneTestCase.SuppressFsync, LuceneTestCase.SuppressReproduceLine, LuceneTestCase.SuppressSysoutChecks, LuceneTestCase.SuppressTempFileChecks, LuceneTestCase.ThrowingConsumer<T>, LuceneTestCase.ThrowingRunnable, LuceneTestCase.Weekly

Field Summary

Fields inherited from class org.apache.lucene.tests.util.LuceneTestCase

assertsAreEnabled, classRules, DEFAULT_LINE_DOCS_FILE, INFOSTREAM, JENKINS_LARGE_LINE_DOCS_FILE, LEAVE_TEMPORARY, MAYBE_CACHE_POLICY, RANDOM_MULTIPLIER, ruleChain, suiteFailureMarker, SYSPROP_AWAITSFIX, SYSPROP_BADAPPLES, SYSPROP_FAILFAST, SYSPROP_MAXFAILURES, SYSPROP_MONSTER, SYSPROP_NIGHTLY, SYSPROP_WEEKLY, TEST_ASSERTS_ENABLED, TEST_AWAITSFIX, TEST_BADAPPLES, TEST_CODEC, TEST_DIRECTORY, TEST_DOCVALUESFORMAT, TEST_LINE_DOCS_FILE, TEST_MONSTER, TEST_NIGHTLY, TEST_POSTINGSFORMAT, TEST_THROTTLING, TEST_WEEKLY, VERBOSE

Constructor Summary

Constructors

Constructor	Description
BaseKnnVectorsFormatTestCase()

Method Summary

Modifier and Type	Method	Description
protected void	addRandomFields(Document doc)	Add random fields to the provided document.
void	init()
static float[]	randomNormalizedVector(int dim)
protected VectorSimilarityFunction	randomSimilarity()
static float[]	randomVector(int dim)
static byte[]	randomVector8(int dim)
protected VectorEncoding	randomVectorEncoding()	This method is overrideable since old codec versions only support VectorEncoding.FLOAT32.
void	testAddIndexesDirectory0()
void	testAddIndexesDirectory01()
void	testAddIndexesDirectory1()
void	testAdvance()
void	testByteVectorScorerIteration()
void	testCheckIndexIncludesVectors()
void	testDeleteAllVectorDocs()
void	testDifferentCodecs1()
void	testDifferentCodecs2()
void	testEmptyByteVectorData()
void	testEmptyFloatVectorData()
void	testFieldConstructor()
void	testFieldConstructorExceptions()
void	testFieldSetValue()
void	testFloatVectorScorerIteration()
void	testIllegalDimChangeTwoDocs()
void	testIllegalDimChangeTwoWriters()
void	testIllegalDimChangeViaAddIndexesCodecReader()
void	testIllegalDimChangeViaAddIndexesDirectory()
void	testIllegalDimChangeViaAddIndexesSlowCodecReader()
void	testIllegalDimensionTooLarge()
void	testIllegalEmptyVector()
void	testIllegalMultipleValues()
void	testIllegalSimilarityFunctionChange()
void	testIllegalSimilarityFunctionChangeTwoWriters()
void	testIllegalSimilarityFunctionChangeViaAddIndexesCodecReader()
void	testIllegalSimilarityFunctionChangeViaAddIndexesDirectory()
void	testIllegalSimilarityFunctionChangeViaAddIndexesSlowCodecReader()
void	testIndexedValueNotAliased()
void	testIndexMultipleKnnVectorFields()
void	testInvalidKnnVectorFieldUsage()
void	testKnnVectorFieldMissingFromOneSegment()
void	testRandom()	Index random vectors, sometimes skipping documents, sometimes deleting a document, sometimes merging, sometimes sorting the index, and verify that the expected values can be read back consistently.
void	testRandomBytes()	Index random vectors as bytes, sometimes skipping documents, sometimes deleting a document, sometimes merging, sometimes sorting the index, and verify that the expected values can be read back consistently.
void	testRandomWithUpdatesAndGraph()	Index random vectors, sometimes skipping documents, sometimes updating a document, sometimes merging, sometimes sorting the index, using an HNSW similarity function so as to also produce a graph, and verify that the expected values can be read back consistently.
void	testSearchWithVisitedLimit()	Tests whether KnnVectorsReader.search(java.lang.String, float[], org.apache.lucene.search.KnnCollector, org.apache.lucene.util.Bits) implementations obey the limit on the number of visited vectors.
void	testSimilarityFunctionIdentifiers()
void	testSortedIndex()
void	testSortedIndexBytes()
void	testSparseVectors()
void	testVectorEncodingOrdinals()
void	testVectorValuesReportCorrectDocs()

Methods inherited from class org.apache.lucene.tests.index.BaseIndexFileFormatTestCase

applyCreatedVersionMajor, excludedExtensionsFromByteCounts, getCodec, getCreatedVersionMajor, maybeWrapWithMergingReader, mergeIsStable, setUp, shouldTestMergeInstance, tearDown, testCheckIntegrityReadsAllBytes, testMergeStability, testMultiClose, testRandomExceptions

Methods inherited from class org.apache.lucene.tests.util.LuceneTestCase

addVirusChecker, assertDeletedDocsEquals, assertDocsAndPositionsEnumEquals, assertDocsEnumEquals, assertDocsSkippingEquals, assertDocValuesEquals, assertDocValuesEquals, assertFieldInfosEquals, assertNormsEquals, assertPointsEquals, assertPositionsSkippingEquals, assertReaderEquals, assertReaderStatisticsEquals, assertStoredFieldEquals, assertStoredFieldsEquals, assertTermsEnumEquals, assertTermsEquals, assertTermsEquals, assertTermsStatisticsEquals, assertTermStatsEquals, assertTermVectorsEquals, asSet, assumeFalse, assumeNoException, assumeTrue, assumeWorkingMMapOnWindows, atLeast, atLeast, callStackContains, callStackContains, callStackContainsAnyOf, closeAfterSuite, closeAfterTest, collate, createTempDir, createTempDir, createTempFile, createTempFile, dumpArray, dumpIterator, ensureSaneIWCOnNightly, expectThrows, expectThrows, expectThrows, expectThrowsAnyOf, expectThrowsAnyOf, getDataInputStream, getDataPath, getJvmForkArguments, getOnlyLeafReader, getTestClass, getTestName, hasWorkingMMapOnWindows, isTestThread, localeForLanguageTag, maybeChangeLiveIndexWriterConfig, maybeWrapReader, newAlcoholicMergePolicy, newAlcoholicMergePolicy, newBytesRef, newBytesRef, newBytesRef, newBytesRef, newBytesRef, newBytesRef, newDirectory, newDirectory, newDirectory, newDirectory, newDirectory, newField, newField, newFSDirectory, newFSDirectory, newIndexWriterConfig, newIndexWriterConfig, newIndexWriterConfig, newIOContext, newIOContext, newLogMergePolicy, newLogMergePolicy, newLogMergePolicy, newLogMergePolicy, newLogMergePolicy, newMaybeVirusCheckingDirectory, newMaybeVirusCheckingFSDirectory, newMergePolicy, newMergePolicy, newMergePolicy, newMockDirectory, newMockDirectory, newMockDirectory, newMockFSDirectory, newMockFSDirectory, newSearcher, newSearcher, newSearcher, newSearcher, newSnapshotIndexWriterConfig, newStringField, newStringField, newStringField, newStringField, newTextField, newTextField, newTieredMergePolicy, newTieredMergePolicy, overrideDefaultQueryCache, overrideTestDefaultQueryCache, random, randomLocale, randomTimeZone, randomVectorFormat, rarely, rarely, replaceMaxFailureRule, resetDefaultQueryCache, restoreCPUCoreCount, restoreIndexWriterMaxDocs, runWithRestrictedPermissions, setIndexWriterMaxDocs, setupCPUCoreCount, setUpExecutorService, shutdownExecutorService, slowFileExists, usually, usually, wrapReader

Methods inherited from class org.junit.Assert

assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertArrayEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertEquals, assertFalse, assertFalse, assertNotEquals, assertNotEquals, assertNotEquals, assertNotEquals, assertNotEquals, assertNotEquals, assertNotEquals, assertNotEquals, assertNotNull, assertNotNull, assertNotSame, assertNotSame, assertNull, assertNull, assertSame, assertSame, assertThat, assertThat, assertThrows, assertThrows, assertTrue, assertTrue, fail, fail

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

BaseKnnVectorsFormatTestCase

public BaseKnnVectorsFormatTestCase()

Method Details

init

public void init()

addRandomFields

protected void addRandomFields(Document doc)

Description copied from class: BaseIndexFileFormatTestCase

Add random fields to the provided document.

Specified by:

addRandomFields in class BaseIndexFileFormatTestCase

testFieldConstructor

public void testFieldConstructor()

testFieldConstructorExceptions

public void testFieldConstructorExceptions()

testFieldSetValue

public void testFieldSetValue()

testIllegalDimChangeTwoDocs

public void testIllegalDimChangeTwoDocs()
                                 throws Exception

Throws:

Exception

testIllegalSimilarityFunctionChange

public void testIllegalSimilarityFunctionChange()
                                         throws Exception

Throws:

Exception

testIllegalDimChangeTwoWriters

public void testIllegalDimChangeTwoWriters()
                                    throws Exception

Throws:

Exception

testIllegalSimilarityFunctionChangeTwoWriters

public void testIllegalSimilarityFunctionChangeTwoWriters()
                                                   throws Exception

Throws:

Exception

testAddIndexesDirectory0

public void testAddIndexesDirectory0()
                              throws Exception

Throws:

Exception

testAddIndexesDirectory1

public void testAddIndexesDirectory1()
                              throws Exception

Throws:

Exception

testAddIndexesDirectory01

public void testAddIndexesDirectory01()
                               throws Exception

Throws:

Exception

testIllegalDimChangeViaAddIndexesDirectory

public void testIllegalDimChangeViaAddIndexesDirectory()
                                                throws Exception

Throws:

Exception

testIllegalSimilarityFunctionChangeViaAddIndexesDirectory

public void testIllegalSimilarityFunctionChangeViaAddIndexesDirectory()
                                                               throws Exception

Throws:

Exception

testIllegalDimChangeViaAddIndexesCodecReader

public void testIllegalDimChangeViaAddIndexesCodecReader()
                                                  throws Exception

Throws:

Exception

testIllegalSimilarityFunctionChangeViaAddIndexesCodecReader

public void testIllegalSimilarityFunctionChangeViaAddIndexesCodecReader()
                                                                 throws Exception

Throws:

Exception

testIllegalDimChangeViaAddIndexesSlowCodecReader

public void testIllegalDimChangeViaAddIndexesSlowCodecReader()
                                                      throws Exception

Throws:

Exception

testIllegalSimilarityFunctionChangeViaAddIndexesSlowCodecReader

public void testIllegalSimilarityFunctionChangeViaAddIndexesSlowCodecReader()
                                                                     throws Exception

Throws:

Exception

testIllegalMultipleValues

public void testIllegalMultipleValues()
                               throws Exception

Throws:

Exception

testIllegalDimensionTooLarge

public void testIllegalDimensionTooLarge()
                                  throws Exception

Throws:

Exception

testIllegalEmptyVector

public void testIllegalEmptyVector()
                            throws Exception

Throws:

Exception

testDifferentCodecs1

public void testDifferentCodecs1()
                          throws Exception

Throws:

Exception

testDifferentCodecs2

public void testDifferentCodecs2()
                          throws Exception

Throws:

Exception

testInvalidKnnVectorFieldUsage

public void testInvalidKnnVectorFieldUsage()

testDeleteAllVectorDocs

public void testDeleteAllVectorDocs()
                             throws Exception

Throws:

Exception

testKnnVectorFieldMissingFromOneSegment

public void testKnnVectorFieldMissingFromOneSegment()
                                             throws Exception

Throws:

Exception

testSparseVectors

public void testSparseVectors()
                       throws Exception

Throws:

Exception

testFloatVectorScorerIteration

public void testFloatVectorScorerIteration()
                                    throws Exception

Throws:

Exception

testByteVectorScorerIteration

public void testByteVectorScorerIteration()
                                   throws Exception

Throws:

Exception

testEmptyFloatVectorData

public void testEmptyFloatVectorData()
                              throws Exception

Throws:

Exception

testEmptyByteVectorData

public void testEmptyByteVectorData()
                             throws Exception

Throws:

Exception

randomSimilarity

protected VectorSimilarityFunction randomSimilarity()

randomVectorEncoding

protected VectorEncoding randomVectorEncoding()

This method is overrideable since old codec versions only support VectorEncoding.FLOAT32.

testIndexedValueNotAliased

public void testIndexedValueNotAliased()
                                throws Exception

Throws:

Exception

testSortedIndex

public void testSortedIndex()
                     throws Exception

Throws:

Exception

testSortedIndexBytes

public void testSortedIndexBytes()
                          throws Exception

Throws:

Exception

testIndexMultipleKnnVectorFields

public void testIndexMultipleKnnVectorFields()
                                      throws Exception

Throws:

Exception

testRandom

public void testRandom()
                throws Exception

Index random vectors, sometimes skipping documents, sometimes deleting a document, sometimes merging, sometimes sorting the index, and verify that the expected values can be read back consistently.

Throws:

Exception

testRandomBytes

public void testRandomBytes()
                     throws Exception

Index random vectors as bytes, sometimes skipping documents, sometimes deleting a document, sometimes merging, sometimes sorting the index, and verify that the expected values can be read back consistently.

Throws:

Exception

testSearchWithVisitedLimit

public void testSearchWithVisitedLimit()
                                throws Exception

Tests whether KnnVectorsReader.search(java.lang.String, float[], org.apache.lucene.search.KnnCollector, org.apache.lucene.util.Bits) implementations obey the limit on the number of visited vectors. This test is a best-effort attempt to capture the right behavior, and isn't meant to define a strict requirement on behavior.

Throws:

Exception

testRandomWithUpdatesAndGraph

public void testRandomWithUpdatesAndGraph()
                                   throws Exception

Index random vectors, sometimes skipping documents, sometimes updating a document, sometimes merging, sometimes sorting the index, using an HNSW similarity function so as to also produce a graph, and verify that the expected values can be read back consistently.

Throws:

Exception

randomVector

public static float[] randomVector(int dim)

randomNormalizedVector

public static float[] randomNormalizedVector(int dim)

randomVector8

public static byte[] randomVector8(int dim)

testCheckIndexIncludesVectors

public void testCheckIndexIncludesVectors()
                                   throws Exception

Throws:

Exception

testSimilarityFunctionIdentifiers

public void testSimilarityFunctionIdentifiers()

testVectorEncodingOrdinals

public void testVectorEncodingOrdinals()

testAdvance

public void testAdvance()
                 throws Exception

Throws:

Exception

testVectorValuesReportCorrectDocs

public void testVectorValuesReportCorrectDocs()
                                       throws Exception

Throws:

Exception