Package com.lancedb.lance.spark.utils

Class VectorUtils

java.lang.Object
com.lancedb.lance.spark.utils.VectorUtils
public class VectorUtils extends Object

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final String
    ARROW_FIXED_SIZE_LIST_SIZE_KEY
     

  • Constructor Summary

    Constructors
    Constructor
    Description
    VectorUtils()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    static String
    createVectorSizePropertyKey(String columnName)
    Create a property key for specifying vector dimensions in table properties.
    static int
    getVectorArrowDimension(org.apache.arrow.vector.types.pojo.Field field)
    Get the vector dimension from an Arrow FixedSizeList field.
    static long
    getVectorDimension(org.apache.spark.sql.types.StructField field)
    Get the vector dimension from a Spark field's metadata.
    static boolean
    isVectorArrowField(org.apache.arrow.vector.types.pojo.Field field)
    Check if an Arrow field is a vector field (FixedSizeList).
    static boolean
    isVectorField(org.apache.spark.sql.types.StructField field)
    Check if a Spark field is a vector field (FixedSizeList) based on its metadata.
    static boolean
    shouldBeFixedSizeList(org.apache.spark.sql.types.DataType dataType, org.apache.spark.sql.types.Metadata metadata)
    Check if a Spark DataType should be treated as a FixedSizeList based on metadata.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

  • Constructor Details

    • VectorUtils

      public VectorUtils()

  • Method Details

    • isVectorField

      public static boolean isVectorField(org.apache.spark.sql.types.StructField field)
      Check if a Spark field is a vector field (FixedSizeList) based on its metadata.
      Parameters:
      field - the Spark struct field to check
      Returns:
      true if the field is a vector field, false otherwise

    • getVectorDimension

      public static long getVectorDimension(org.apache.spark.sql.types.StructField field)
      Get the vector dimension from a Spark field's metadata.
      Parameters:
      field - the Spark struct field
      Returns:
      the vector dimension, or -1 if not a vector field

    • isVectorArrowField

      public static boolean isVectorArrowField(org.apache.arrow.vector.types.pojo.Field field)
      Check if an Arrow field is a vector field (FixedSizeList).
      Parameters:
      field - the Arrow field to check
      Returns:
      true if the field is a vector field, false otherwise

    • getVectorArrowDimension

      public static int getVectorArrowDimension(org.apache.arrow.vector.types.pojo.Field field)
      Get the vector dimension from an Arrow FixedSizeList field.
      Parameters:
      field - the Arrow field
      Returns:
      the vector dimension, or -1 if not a vector field

    • shouldBeFixedSizeList

      public static boolean shouldBeFixedSizeList(org.apache.spark.sql.types.DataType dataType, org.apache.spark.sql.types.Metadata metadata)
      Check if a Spark DataType should be treated as a FixedSizeList based on metadata.
      Parameters:
      dataType - the Spark data type
      metadata - the field metadata
      Returns:
      true if it should be a FixedSizeList, false otherwise

    • createVectorSizePropertyKey

      public static String createVectorSizePropertyKey(String columnName)
      Create a property key for specifying vector dimensions in table properties. This is used in CREATE TABLE statements.
      Parameters:
      columnName - the name of the column
      Returns:
      the property key for specifying vector dimension