com.lancedb.lance

Class Dataset

java.lang.Object

com.lancedb.lance.Dataset

All Implemented Interfaces:

Closeable, AutoCloseable

public class Dataset
extends Object
implements Closeable

Class representing a Lance dataset, interfacing with the native lance library. This class provides functionality to open and manage datasets with native code. The native library is loaded statically and utilized through native methods. It implements the Closeable interface to ensure proper resource management.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	Dataset.Tags Tag operations of the dataset.

Method Summary

Modifier and Type	Method and Description
void	addColumns(org.apache.arrow.c.ArrowArrayStream stream, Optional<Long> batchSize) Add columns to the dataset.
void	addColumns(List<org.apache.arrow.vector.types.pojo.Field> fields) Add columns to the dataset.
void	addColumns(org.apache.arrow.vector.types.pojo.Schema schema) Add columns to the dataset.
void	addColumns(SqlExpressions sqlExpressions, Optional<Long> batchSize) Add columns to the dataset.
org.apache.arrow.memory.BufferAllocator	allocator()
void	alterColumns(List<ColumnAlteration> columnAlterations) Alter columns in the dataset.
long	calculateDataSize() Calculate the size of the dataset.
void	checkoutLatest() Checkout the dataset to the latest version.
Dataset	checkoutTag(String tag) Checks out a specific tag of the dataset.
Dataset	checkoutVersion(long version) Checks out a specific version of the dataset.
void	close() Closes this dataset and releases any system resources associated with it.
boolean	closed() Checks if the dataset is closed.
static Dataset	commit(org.apache.arrow.memory.BufferAllocator allocator, String path, FragmentOperation operation, Optional<Long> readVersion) Deprecated.
static Dataset	commit(org.apache.arrow.memory.BufferAllocator allocator, String path, FragmentOperation operation, Optional<Long> readVersion, Map<String,String> storageOptions) Deprecated.
static Dataset	commitAppend(String path, Optional<Long> readVersion, List<FragmentMetadata> fragmentsMetadata, Map<String,String> storageOptions) Deprecated.
static Dataset	commitOverwrite(String path, long arrowSchemaMemoryAddress, Optional<Long> readVersion, List<FragmentMetadata> fragmentsMetadata, Map<String,String> storageOptions) Deprecated.
Dataset	commitTransaction(Transaction transaction) Commit a single transaction and return a new Dataset with the new version.
long	countRows() Count the number of rows in the dataset.
long	countRows(String filter) Count the number of rows in the dataset.
static Dataset	create(org.apache.arrow.memory.BufferAllocator allocator, org.apache.arrow.c.ArrowArrayStream stream, String path, WriteParams params) Create a dataset with given stream.
static Dataset	create(org.apache.arrow.memory.BufferAllocator allocator, String path, org.apache.arrow.vector.types.pojo.Schema schema, WriteParams params) Creates an empty dataset.
void	createIndex(List<String> columns, IndexType indexType, Optional<String> name, IndexParams params, boolean replace) Creates a new index on the dataset.
void	delete(String predicate) Delete rows of data by predicate.
void	deleteConfigKeys(Set<String> deleteKeys) Delete the config keys of the dataset.
static void	drop(String path, Map<String,String> storageOptions) Drop a Dataset.
void	dropColumns(List<String> columns) Drop columns from the dataset.
Map<String,String>	getConfig() Get the table config of the dataset.
Fragment	getFragment(int fragmentId)
List<Fragment>	getFragments() Get all fragments in this dataset.
LanceSchema	getLanceSchema() Get the LanceSchema of the dataset with field ids.
org.apache.arrow.vector.types.pojo.Schema	getSchema() Gets the arrow schema of the dataset.
Version	getVersion() Gets the currently checked out version of the dataset.
long	latestVersion()
List<String>	listIndexes()
List<Version>	listVersions() Get the version history of the dataset.
LanceScanner	newScan() Create a new Dataset Scanner.
LanceScanner	newScan(long batchSize) Create a new Dataset Scanner.
LanceScanner	newScan(ScanOptions options) Create a new Dataset Scanner.
Transaction.Builder	newTransactionBuilder() Create a new transaction builder at current version for the dataset.
static Dataset	open(org.apache.arrow.memory.BufferAllocator allocator, String path, ReadOptions options) Open a dataset from the specified path with additional options.
static Dataset	open(String path) Open a dataset from the specified path.
static Dataset	open(String path, org.apache.arrow.memory.BufferAllocator allocator) Open a dataset from the specified path.
static Dataset	open(String path, ReadOptions options) Open a dataset from the specified path.
Optional<Transaction>	readTransaction() Get the Transaction of the dataset at the current version.
void	replaceFieldMetadata(Map<Integer,Map<String,String>> fieldMetadataMap) Replace target field metadata of the dataset.
void	replaceSchemaMetadata(Map<String,String> metadata) Replace the schema metadata of the dataset.
void	restore() Restore the currently checked out version of the dataset as the latest version.
SqlQuery	sql(String sql) Execute SQL query on the dataset.
Dataset.Tags	tags() Returns a Dataset.Tags instance for performing tag-related operations on the dataset.
org.apache.arrow.vector.ipc.ArrowReader	take(List<Long> indices, List<String> columns) Select rows of data by index.
void	updateConfig(Map<String,String> tableConfig) Update the config of the dataset.
String	uri() Gets the URI of the dataset.
long	version() Get the currently checked out version id of the dataset

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

create

public static Dataset create(org.apache.arrow.memory.BufferAllocator allocator,
                             String path,
                             org.apache.arrow.vector.types.pojo.Schema schema,
                             WriteParams params)

Creates an empty dataset.

Parameters:

allocator - the buffer allocator

path - dataset uri

schema - dataset schema

params - write params

Returns:

Dataset

create

public static Dataset create(org.apache.arrow.memory.BufferAllocator allocator,
                             org.apache.arrow.c.ArrowArrayStream stream,
                             String path,
                             WriteParams params)

Create a dataset with given stream.

Parameters:

allocator - buffer allocator

stream - arrow stream

path - dataset uri

params - write parameters

Returns:

Dataset

open

public static Dataset open(String path)

Open a dataset from the specified path.

Parameters:

path - file path

Returns:

Dataset

open

public static Dataset open(String path,
                           ReadOptions options)

Open a dataset from the specified path.

Parameters:

path - file path

options - the open options

Returns:

Dataset

open

public static Dataset open(String path,
                           org.apache.arrow.memory.BufferAllocator allocator)

Open a dataset from the specified path.

Parameters:

path - file path

allocator - Arrow buffer allocator

Returns:

Dataset

open

public static Dataset open(org.apache.arrow.memory.BufferAllocator allocator,
                           String path,
                           ReadOptions options)

Open a dataset from the specified path with additional options.

Parameters:

allocator - Arrow buffer allocator

path - file path

options - the open options

Returns:

Dataset

commit

@Deprecated
public static Dataset commit(org.apache.arrow.memory.BufferAllocator allocator,
                                         String path,
                                         FragmentOperation operation,
                                         Optional<Long> readVersion)

Deprecated.

Create a new version of dataset. Use Transaction instead

Parameters:

allocator - the buffer allocator

path - The file path of the dataset to open.

operation - The operation to apply to the dataset.

readVersion - The version of the dataset that was used as the base for the changes. This is not needed for overwrite or restore operations.

Returns:

A new instance of Dataset linked to the opened dataset.

commit

@Deprecated
public static Dataset commit(org.apache.arrow.memory.BufferAllocator allocator,
                                         String path,
                                         FragmentOperation operation,
                                         Optional<Long> readVersion,
                                         Map<String,String> storageOptions)

Deprecated.

commitAppend

@Deprecated
public static Dataset commitAppend(String path,
                                               Optional<Long> readVersion,
                                               List<FragmentMetadata> fragmentsMetadata,
                                               Map<String,String> storageOptions)

Deprecated.

Use Transaction instead

commitOverwrite

@Deprecated
public static Dataset commitOverwrite(String path,
                                                  long arrowSchemaMemoryAddress,
                                                  Optional<Long> readVersion,
                                                  List<FragmentMetadata> fragmentsMetadata,
                                                  Map<String,String> storageOptions)

Deprecated.

Use Transaction instead

allocator

public org.apache.arrow.memory.BufferAllocator allocator()

newTransactionBuilder

public Transaction.Builder newTransactionBuilder()

Create a new transaction builder at current version for the dataset. The dataset itself will not refresh after the transaction committed.

Returns:

A new instance of Transaction.Builder linked to the opened dataset.

commitTransaction

public Dataset commitTransaction(Transaction transaction)

Commit a single transaction and return a new Dataset with the new version. Original dataset version will not be refreshed.

Parameters:

transaction - The transaction to commit

Returns:

A new instance of Dataset linked to committed version.

drop

public static void drop(String path,
                        Map<String,String> storageOptions)

Drop a Dataset.

Parameters:

path - The file path of the dataset

storageOptions - Storage options

addColumns

public void addColumns(SqlExpressions sqlExpressions,
                       Optional<Long> batchSize)

Add columns to the dataset.

Parameters:

sqlExpressions - The SQL expressions to add columns

batchSize - The number of rows to read at a time from the source dataset when applying the transform.

addColumns

public void addColumns(org.apache.arrow.c.ArrowArrayStream stream,
                       Optional<Long> batchSize)

Add columns to the dataset.

Parameters:

stream - The Arrow Array Stream generated by arrow reader to add columns.

batchSize - The number of rows to read at a time from the source dataset when applying the transform.

addColumns

public void addColumns(org.apache.arrow.vector.types.pojo.Schema schema)

Add columns to the dataset.

Parameters:

schema - The Arrow schema definitions to add columns.

addColumns

public void addColumns(List<org.apache.arrow.vector.types.pojo.Field> fields)

Add columns to the dataset.

Parameters:

fields - The Arrow field definitions to add columns.

dropColumns

public void dropColumns(List<String> columns)

Drop columns from the dataset.

Parameters:

columns - The columns to drop

alterColumns

public void alterColumns(List<ColumnAlteration> columnAlterations)

Alter columns in the dataset.

Parameters:

columnAlterations - The list of columns need to be altered.

newScan

public LanceScanner newScan()

Create a new Dataset Scanner.

Returns:

a dataset scanner

newScan

public LanceScanner newScan(long batchSize)

Create a new Dataset Scanner.

Parameters:

batchSize - the scan options with batch size, columns filter, and substrait

Returns:

a dataset scanner

newScan

public LanceScanner newScan(ScanOptions options)

Create a new Dataset Scanner.

Parameters:

options - the scan options

Returns:

a dataset scanner

take

public org.apache.arrow.vector.ipc.ArrowReader take(List<Long> indices,
                                                    List<String> columns)
                                             throws IOException

Select rows of data by index.

Parameters:

indices - the indices to take

columns - the columns to take

Returns:

an ArrowReader

Throws:

IOException

delete

public void delete(String predicate)

Delete rows of data by predicate.

Parameters:

predicate - the predicate to delete

uri

public String uri()

Gets the URI of the dataset.

Returns:

the URI of the dataset

version

public long version()

Get the currently checked out version id of the dataset

Returns:

the version id of the dataset

getVersion

public Version getVersion()

Gets the currently checked out version of the dataset.

Returns:

the version of the dataset

listVersions

public List<Version> listVersions()

Get the version history of the dataset.

Returns:

the version history of the dataset

latestVersion

public long latestVersion()

Returns:

the latest version of the dataset.

checkoutLatest

public void checkoutLatest()

Checkout the dataset to the latest version.

checkoutVersion

public Dataset checkoutVersion(long version)

Checks out a specific version of the dataset. If the version is already checked out, it returns a new Java Dataset object pointing to the same underlying Rust Dataset object

Parameters:

version - the version to check out

Returns:

a new Dataset instance with the specified version checked out

checkoutTag

public Dataset checkoutTag(String tag)

Checks out a specific tag of the dataset. If the underlying version is already checked out, it returns a new Java Dataset object pointing to the same underlying Rust Dataset object

Parameters:

tag - the tag to check out

Returns:

a new Dataset instance with the specified tag checked out

restore

public void restore()

Restore the currently checked out version of the dataset as the latest version. This operation produces a new version and doesn't influence any old versions and tags.

createIndex

public void createIndex(List<String> columns,
                        IndexType indexType,
                        Optional<String> name,
                        IndexParams params,
                        boolean replace)

Creates a new index on the dataset. Only vector indexes are supported.

Parameters:

columns - the columns to index from

indexType - the index type

name - the name of the created index

params - index params

replace - whether to replace the existing index

countRows

public long countRows()

Count the number of rows in the dataset.

Returns:

num of rows

countRows

public long countRows(String filter)

Count the number of rows in the dataset.

Parameters:

filter - the filter expr to count row

Returns:

num of rows

calculateDataSize

public long calculateDataSize()

Calculate the size of the dataset.

Returns:

the size of the dataset

getFragments

public List<Fragment> getFragments()

Get all fragments in this dataset.

Returns:

A list of Fragment.

getSchema

public org.apache.arrow.vector.types.pojo.Schema getSchema()

Gets the arrow schema of the dataset.

Returns:

the arrow schema

getLanceSchema

public LanceSchema getLanceSchema()

Get the LanceSchema of the dataset with field ids.

Returns:

the LanceSchema

readTransaction

public Optional<Transaction> readTransaction()

Get the Transaction of the dataset at the current version.

Returns:

the Transaction

listIndexes

public List<String> listIndexes()

Returns:

all the created indexes names

getConfig

public Map<String,String> getConfig()

Get the table config of the dataset.

Returns:

the table config

updateConfig

public void updateConfig(Map<String,String> tableConfig)

Update the config of the dataset. This operation will only overwrite and NOT delete the existing config.

Parameters:

tableConfig - the config to update

deleteConfigKeys

public void deleteConfigKeys(Set<String> deleteKeys)

Delete the config keys of the dataset.

Parameters:

deleteKeys - the config keys to delete

close

public void close()

Closes this dataset and releases any system resources associated with it. If the dataset is already closed, then invoking this method has no effect.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

closed

public boolean closed()

Checks if the dataset is closed.

Returns:

true if the dataset is closed, false otherwise.

getFragment

public Fragment getFragment(int fragmentId)

tags

public Dataset.Tags tags()

Returns a Dataset.Tags instance for performing tag-related operations on the dataset.

Returns:

new Tags instance for dataset tag operations

See Also:

Dataset.Tags

replaceSchemaMetadata

public void replaceSchemaMetadata(Map<String,String> metadata)

Replace the schema metadata of the dataset.

Parameters:

metadata - the new table metadata

replaceFieldMetadata

public void replaceFieldMetadata(Map<Integer,Map<String,String>> fieldMetadataMap)

Replace target field metadata of the dataset. This method won't affect fields not in the map

Parameters:

fieldMetadataMap - field id to metadata map

sql

public SqlQuery sql(String sql)

Execute SQL query on the dataset. The underlying SQL engine is DataFusion. Please refer to the DataFusion documentation for supported SQL syntax.

Parameters:

sql - SELECT statement to execute. The default FROM table name is `dataset`, for example: SELECT * FROM `dataset` LIMIT 10. If FROM table name is a custom value, the SqlQuery.tableName(String) should be invoked to set the custom table name.

Returns:

a SqlQuery instance.