com.lancedb.lance.index.vector

Class IvfBuildParams.Builder

java.lang.Object

com.lancedb.lance.index.vector.IvfBuildParams.Builder

Enclosing class:

IvfBuildParams

public static class IvfBuildParams.Builder
extends Object

Constructor Summary

Constructors

Constructor and Description
Builder() Parameters for building an IVF index.

Method Summary

Modifier and Type	Method and Description
IvfBuildParams	build()
IvfBuildParams.Builder	setMaxIters(int maxIters)
IvfBuildParams.Builder	setNumPartitions(int numPartitions)
IvfBuildParams.Builder	setSampleRate(int sampleRate) Set the sample rate for training IVF centroids IVF centroids are trained from a sample of the data (determined by the sample_rate).
IvfBuildParams.Builder	setShufflePartitionBatches(int shufflePartitionBatches) Sets the number of batches, using the row group size of the dataset, to include in each shuffle partition.
IvfBuildParams.Builder	setShufflePartitionConcurrency(int shufflePartitionConcurrency) Set the number of shuffle partitions to process concurrently.
IvfBuildParams.Builder	setUseResidual(boolean useResidual) Set whether to use residual for k-means clustering.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Builder

public Builder()

Parameters for building an IVF index. Train IVF centroids for the given vector column. This will run k-means clustering on the given vector column to train the IVF centroids. This is the first step in several vector indices. The centroids will be used to partition the vectors into different clusters. IVF centroids are trained from a sample of the data (determined by the sample_rate). While this sample is not huge it might still be quite large.

Method Detail

setNumPartitions

public IvfBuildParams.Builder setNumPartitions(int numPartitions)

Parameters:

numPartitions - set the number of partitions of IVF (Inverted File Index) Default to 32

Returns:

Builder

setMaxIters

public IvfBuildParams.Builder setMaxIters(int maxIters)

Parameters:

maxIters - set the maximum number of iterations for k-means clustering. Default to 50.

Returns:

Builder

setSampleRate

public IvfBuildParams.Builder setSampleRate(int sampleRate)

Set the sample rate for training IVF centroids IVF centroids are trained from a sample of the data (determined by the sample_rate). While this sample is not huge it might still be quite large. Default to 256.

Parameters:

sampleRate - set the sample rate for training IVF centroids

Returns:

Builder

setShufflePartitionBatches

public IvfBuildParams.Builder setShufflePartitionBatches(int shufflePartitionBatches)

Sets the number of batches, using the row group size of the dataset, to include in each shuffle partition. Default value is 10240. Assuming the row group size is 1024, each shuffle partition will hold 10240 * 1024 = 10,485,760 rows. By making this value smaller, this shuffle will consume less memory but will take longer to complete, and vice versa.

Parameters:

shufflePartitionBatches - the number of batches to include in shuffle

Returns:

Builder

setShufflePartitionConcurrency

public IvfBuildParams.Builder setShufflePartitionConcurrency(int shufflePartitionConcurrency)

Set the number of shuffle partitions to process concurrently. Default value is 2. By making this value smaller, this shuffle will consume less memory but will take longer to complete, and vice versa.

Parameters:

shufflePartitionConcurrency - the number of shuffle partitions to process concurrently

Returns:

Builder

setUseResidual

public IvfBuildParams.Builder setUseResidual(boolean useResidual)

Set whether to use residual for k-means clustering. Default value is true.

Parameters:

useResidual - whether to use residual for k-means clustering

Returns:

Builder

build

public IvfBuildParams build()