Package org.apache.lucene.internal.vectorization

Interface VectorUtilSupport

public interface VectorUtilSupport
Interface for implementations of VectorUtil support.
NOTE: This API is for internal purposes only and might change in incompatible ways in the next release.

  • Method Summary

    Modifier and Type
    Method
    Description
    float
    cosine(byte[] a, byte[] b)
    Returns the cosine similarity between the two byte vectors.
    float
    cosine(float[] v1, float[] v2)
    Returns the cosine similarity between the two vectors.
    int
    dotProduct(byte[] a, byte[] b)
    Returns the dot product computed over signed bytes.
    float
    dotProduct(float[] a, float[] b)
    Calculates the dot product of the given float arrays.
    int
    findNextGEQ(int[] buffer, int target, int from, int to)
    Given an array buffer that is sorted between indexes 0 inclusive and to exclusive, find the first array index whose value is greater than or equal to target.
    long
    int4BitDotProduct(byte[] int4Quantized, byte[] binaryQuantized)
    Compute the dot product between a quantized int4 vector and a binary quantized vector.
    int
    int4DotProduct(byte[] a, boolean apacked, byte[] b, boolean bpacked)
    Returns the dot product over the computed bytes, assuming the values are int4 encoded.
    float
    minMaxScalarQuantize(float[] vector, byte[] dest, float scale, float alpha, float minQuantile, float maxQuantile)
    Quantizes vector, putting the result into dest.
    float
    recalculateScalarQuantizationOffset(byte[] vector, float oldAlpha, float oldMinQuantile, float scale, float alpha, float minQuantile, float maxQuantile)
    Recalculates the offset for vector.
    int
    squareDistance(byte[] a, byte[] b)
    Returns the sum of squared differences of the two byte vectors.
    float
    squareDistance(float[] a, float[] b)
    Returns the sum of squared differences of the two vectors.

  • Method Details

    • dotProduct

      float dotProduct(float[] a, float[] b)
      Calculates the dot product of the given float arrays.

    • cosine

      float cosine(float[] v1, float[] v2)
      Returns the cosine similarity between the two vectors.

    • squareDistance

      float squareDistance(float[] a, float[] b)
      Returns the sum of squared differences of the two vectors.

    • dotProduct

      int dotProduct(byte[] a, byte[] b)
      Returns the dot product computed over signed bytes.

    • int4DotProduct

      int int4DotProduct(byte[] a, boolean apacked, byte[] b, boolean bpacked)
      Returns the dot product over the computed bytes, assuming the values are int4 encoded.

    • cosine

      float cosine(byte[] a, byte[] b)
      Returns the cosine similarity between the two byte vectors.

    • squareDistance

      int squareDistance(byte[] a, byte[] b)
      Returns the sum of squared differences of the two byte vectors.

    • findNextGEQ

      int findNextGEQ(int[] buffer, int target, int from, int to)
      Given an array buffer that is sorted between indexes 0 inclusive and to exclusive, find the first array index whose value is greater than or equal to target. This index is guaranteed to be at least from. If there is no such array index, to is returned.

    • int4BitDotProduct

      long int4BitDotProduct(byte[] int4Quantized, byte[] binaryQuantized)
      Compute the dot product between a quantized int4 vector and a binary quantized vector. It is assumed that the int4 quantized bits are packed in the byte array in the same way as the OptimizedScalarQuantizer.transposeHalfByte(byte[], byte[]) and that the binary bits are packed the same way as OptimizedScalarQuantizer.packAsBinary(byte[], byte[]).
      Parameters:
      int4Quantized - half byte packed int4 quantized vector
      binaryQuantized - byte packed binary quantized vector
      Returns:
      the dot product

    • minMaxScalarQuantize

      float minMaxScalarQuantize(float[] vector, byte[] dest, float scale, float alpha, float minQuantile, float maxQuantile)
      Quantizes vector, putting the result into dest.
      Parameters:
      vector - the vector to quantize
      dest - the destination vector
      scale - the scaling factor
      alpha - the alpha value
      minQuantile - the lower quantile of the distribution
      maxQuantile - the upper quantile of the distribution
      Returns:
      the corrective offset that needs to be applied to the score

    • recalculateScalarQuantizationOffset

      float recalculateScalarQuantizationOffset(byte[] vector, float oldAlpha, float oldMinQuantile, float scale, float alpha, float minQuantile, float maxQuantile)
      Recalculates the offset for vector.
      Parameters:
      vector - the vector to quantize
      oldAlpha - the previous alpha value
      oldMinQuantile - the previous lower quantile
      scale - the scaling factor
      alpha - the alpha value
      minQuantile - the lower quantile of the distribution
      maxQuantile - the upper quantile of the distribution
      Returns:
      the new corrective offset