Dataset (Lance Core 0.38.3 API)

java.lang.Object
- com.lancedb.lance.Dataset

All Implemented Interfaces:

Closeable, AutoCloseable
```
public class Dataset
extends Object
implements Closeable
```
Class representing a Lance dataset, interfacing with the native lance library. This class provides functionality to open and manage datasets with native code. The native library is loaded statically and utilized through native methods. It implements the Closeable interface to ensure proper resource management.

Nested Class Summary

Nested Classes
Modifier and Type Class and Description

class Dataset.Branches
Branch operations of the dataset.

class Dataset.Tags
Tag operations of the dataset.

Nested Classes
Modifier and Type	Class and Description
`class`	`Dataset.Branches` Branch operations of the dataset.
`class`	`Dataset.Tags` Tag operations of the dataset.

Method Summary

All Methods Static Methods Instance Methods Concrete Methods Deprecated Methods
Modifier and Type	Method and Description
`void`	`addColumns(org.apache.arrow.c.ArrowArrayStream stream, Optional<Long> batchSize)` Add columns to the dataset.
`void`	`addColumns(List<org.apache.arrow.vector.types.pojo.Field> fields)` Add columns to the dataset.
`void`	`addColumns(org.apache.arrow.vector.types.pojo.Schema schema)` Add columns to the dataset.
`void`	`addColumns(SqlExpressions sqlExpressions, Optional<Long> batchSize)` Add columns to the dataset.
`org.apache.arrow.memory.BufferAllocator`	`allocator()`
`void`	`alterColumns(List<ColumnAlteration> columnAlterations)` Alter columns in the dataset.
`Dataset.Branches`	`branches()` Branch operations aligned with Rust's Dataset branch APIs.
`long`	`calculateDataSize()` Calculate the size of the dataset.
`Dataset`	`checkout(Ref ref)` Checkout using a unified `Ref` which can be a tag, the latest version on main/branch or a specified (branch_name, version_number).
`void`	`checkoutLatest()` Checkout the dataset to the latest version.
`Dataset`	`checkoutTag(String tag)` Checks out a specific tag of the dataset.
`Dataset`	`checkoutVersion(long version)` Checks out a specific version of the dataset.
`void`	`close()` Closes this dataset and releases any system resources associated with it.
`boolean`	`closed()` Checks if the dataset is closed.
`static Dataset`	`commit(org.apache.arrow.memory.BufferAllocator allocator, String path, FragmentOperation operation, Optional<Long> readVersion)` Deprecated.
`static Dataset`	`commit(org.apache.arrow.memory.BufferAllocator allocator, String path, FragmentOperation operation, Optional<Long> readVersion, Map<String,String> storageOptions)` Deprecated.
`static Dataset`	`commitAppend(String path, Optional<Long> readVersion, List<FragmentMetadata> fragmentsMetadata, Map<String,String> storageOptions)` Deprecated.
`static Dataset`	`commitOverwrite(String path, long arrowSchemaMemoryAddress, Optional<Long> readVersion, List<FragmentMetadata> fragmentsMetadata, Map<String,String> storageOptions)` Deprecated.
`Dataset`	`commitTransaction(Transaction transaction)` Commit a single transaction and return a new Dataset with the new version.
`void`	`compact()` Compact the dataset with default options.
`void`	`compact(CompactionOptions options)` Compact the dataset to improve performance.
`long`	`countRows()` Count the number of rows in the dataset.
`long`	`countRows(String filter)` Count the number of rows in the dataset.
`static Dataset`	`create(org.apache.arrow.memory.BufferAllocator allocator, org.apache.arrow.c.ArrowArrayStream stream, String path, WriteParams params)` Create a dataset with given stream.
`static Dataset`	`create(org.apache.arrow.memory.BufferAllocator allocator, String path, org.apache.arrow.vector.types.pojo.Schema schema, WriteParams params)` Creates an empty dataset.
`void`	`createIndex(List<String> columns, IndexType indexType, Optional<String> name, IndexParams params, boolean replace)` Creates a new index on the dataset.
`void`	`delete(String predicate)` Delete rows of data by predicate.
`void`	`deleteConfigKeys(Set<String> deleteKeys)` Deprecated. Use `newTransactionBuilder()` with `UpdateConfig` operation instead
`static void`	`drop(String path, Map<String,String> storageOptions)` Drop a Dataset.
`void`	`dropColumns(List<String> columns)` Drop columns from the dataset.
`Map<String,String>`	`getConfig()` Get the table config of the dataset.
`Fragment`	`getFragment(int fragmentId)`
`List<Fragment>`	`getFragments()` Get all fragments in this dataset.
`LanceSchema`	`getLanceSchema()` Get the `LanceSchema` of the dataset with field ids.
`org.apache.arrow.vector.types.pojo.Schema`	`getSchema()` Gets the arrow schema of the dataset.
`Map<String,String>`	`getTableMetadata()` Get the table metadata of the dataset.
`Version`	`getVersion()` Gets the currently checked out version of the dataset.
`long`	`latestVersion()`
`List<String>`	`listIndexes()`
`List<Version>`	`listVersions()` Get the version history of the dataset.
`MergeInsertResult`	`mergeInsert(MergeInsertParams mergeInsert, org.apache.arrow.c.ArrowArrayStream source)` Merge source data with the existing target data.
`LanceScanner`	`newScan()` Create a new Dataset Scanner.
`LanceScanner`	`newScan(long batchSize)` Create a new Dataset Scanner.
`LanceScanner`	`newScan(ScanOptions options)` Create a new Dataset Scanner.
`Transaction.Builder`	`newTransactionBuilder()` Create a new transaction builder at current version for the dataset.
`static Dataset`	`open(org.apache.arrow.memory.BufferAllocator allocator, String path, ReadOptions options)` Open a dataset from the specified path with additional options.
`static Dataset`	`open(String path)` Open a dataset from the specified path.
`static Dataset`	`open(String path, org.apache.arrow.memory.BufferAllocator allocator)` Open a dataset from the specified path.
`static Dataset`	`open(String path, ReadOptions options)` Open a dataset from the specified path.
`Optional<Transaction>`	`readTransaction()` Get the `Transaction` of the dataset at the current version.
`void`	`restore()` Restore the currently checked out version of the dataset as the latest version.
`Dataset`	`shallowClone(String targetPath, Ref ref)`
`Dataset`	`shallowClone(String targetPath, Ref ref, Map<String,String> storageOptions)` Shallow clone the specified tag into a new dataset at the target path.
`SqlQuery`	`sql(String sql)` Execute SQL query on the dataset.
`Dataset.Tags`	`tags()` Returns a `Dataset.Tags` instance for performing tag-related operations on the dataset.
`org.apache.arrow.vector.ipc.ArrowReader`	`take(List<Long> indices, List<String> columns)` Select rows of data by index.
`List<BlobFile>`	`takeBlobs(List<Long> rowIds, String column)` Open blob files for given row ids on a blob column.
`List<BlobFile>`	`takeBlobsByIndices(List<Long> rowIndices, String column)` Open blob files for given row indices on a blob column.
`void`	`updateConfig(Map<String,String> tableConfig)` Deprecated. Use `newTransactionBuilder()` with `UpdateConfig` operation instead
`String`	`uri()` Gets the URI of the dataset.
`long`	`version()` Get the currently checked out version id of the dataset

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Method Detail
  - create
```
public static Dataset create(org.apache.arrow.memory.BufferAllocator allocator,
                             String path,
                             org.apache.arrow.vector.types.pojo.Schema schema,
                             WriteParams params)
```
    Creates an empty dataset.
    
    Parameters:
    
    allocator - the buffer allocator
    
    path - dataset uri
    
    schema - dataset schema
    
    params - write params
    
    Returns:
    
    Dataset
  - create
```
public static Dataset create(org.apache.arrow.memory.BufferAllocator allocator,
                             org.apache.arrow.c.ArrowArrayStream stream,
                             String path,
                             WriteParams params)
```
    Create a dataset with given stream.
    
    Parameters:
    
    allocator - buffer allocator
    
    stream - arrow stream
    
    path - dataset uri
    
    params - write parameters
    
    Returns:
    
    Dataset
  - open
```
public static Dataset open(String path)
```
    Open a dataset from the specified path.
    
    Parameters:
    
    path - file path
    
    Returns:
    
    Dataset
  - open
```
public static Dataset open(String path,
                           ReadOptions options)
```
    Open a dataset from the specified path.
    
    Parameters:
    
    path - file path
    
    options - the open options
    
    Returns:
    
    Dataset
  - open
```
public static Dataset open(String path,
                           org.apache.arrow.memory.BufferAllocator allocator)
```
    Open a dataset from the specified path.
    
    Parameters:
    
    path - file path
    
    allocator - Arrow buffer allocator
    
    Returns:
    
    Dataset
  - open
```
public static Dataset open(org.apache.arrow.memory.BufferAllocator allocator,
                           String path,
                           ReadOptions options)
```
    Open a dataset from the specified path with additional options.
    
    Parameters:
    
    allocator - Arrow buffer allocator
    
    path - file path
    
    options - the open options
    
    Returns:
    
    Dataset
  - commit
```
@Deprecated
public static Dataset commit(org.apache.arrow.memory.BufferAllocator allocator,
                                         String path,
                                         FragmentOperation operation,
                                         Optional<Long> readVersion)
```
    Deprecated.
    
    Create a new version of dataset. Use Transaction instead
    
    Parameters:
    
    allocator - the buffer allocator
    
    path - The file path of the dataset to open.
    
    operation - The operation to apply to the dataset.
    
    readVersion - The version of the dataset that was used as the base for the changes. This is not needed for overwrite or restore operations.
    
    Returns:
    
    A new instance of Dataset linked to the opened dataset.
  - commit
```
@Deprecated
public static Dataset commit(org.apache.arrow.memory.BufferAllocator allocator,
                                         String path,
                                         FragmentOperation operation,
                                         Optional<Long> readVersion,
                                         Map<String,String> storageOptions)
```
    Deprecated.
  - commitAppend
```
@Deprecated
public static Dataset commitAppend(String path,
                                               Optional<Long> readVersion,
                                               List<FragmentMetadata> fragmentsMetadata,
                                               Map<String,String> storageOptions)
```
    Deprecated.
    
    Use Transaction instead
  - commitOverwrite
```
@Deprecated
public static Dataset commitOverwrite(String path,
                                                  long arrowSchemaMemoryAddress,
                                                  Optional<Long> readVersion,
                                                  List<FragmentMetadata> fragmentsMetadata,
                                                  Map<String,String> storageOptions)
```
    Deprecated.
    
    Use Transaction instead
  - allocator
```
public org.apache.arrow.memory.BufferAllocator allocator()
```
  - newTransactionBuilder
```
public Transaction.Builder newTransactionBuilder()
```
    Create a new transaction builder at current version for the dataset. The dataset itself will not refresh after the transaction committed.
    
    Returns:
    
    A new instance of Transaction.Builder linked to the opened dataset.
  - commitTransaction
```
public Dataset commitTransaction(Transaction transaction)
```
    Commit a single transaction and return a new Dataset with the new version. Original dataset version will not be refreshed.
    
    Parameters:
    
    transaction - The transaction to commit
    
    Returns:
    
    A new instance of Dataset linked to committed version.
  - drop
```
public static void drop(String path,
                        Map<String,String> storageOptions)
```
    Drop a Dataset.
    
    Parameters:
    
    path - The file path of the dataset
    
    storageOptions - Storage options
  - addColumns
```
public void addColumns(SqlExpressions sqlExpressions,
                       Optional<Long> batchSize)
```
    Add columns to the dataset.
    
    Parameters:
    
    sqlExpressions - The SQL expressions to add columns
    
    batchSize - The number of rows to read at a time from the source dataset when applying the transform.
  - addColumns
```
public void addColumns(org.apache.arrow.c.ArrowArrayStream stream,
                       Optional<Long> batchSize)
```
    Add columns to the dataset.
    
    Parameters:
    
    stream - The Arrow Array Stream generated by arrow reader to add columns.
    
    batchSize - The number of rows to read at a time from the source dataset when applying the transform.
  - addColumns
```
public void addColumns(org.apache.arrow.vector.types.pojo.Schema schema)
```
    Add columns to the dataset.
    
    Parameters:
    
    schema - The Arrow schema definitions to add columns.
  - addColumns
```
public void addColumns(List<org.apache.arrow.vector.types.pojo.Field> fields)
```
    Add columns to the dataset.
    
    Parameters:
    
    fields - The Arrow field definitions to add columns.
  - dropColumns
```
public void dropColumns(List<String> columns)
```
    Drop columns from the dataset.
    
    Parameters:
    
    columns - The columns to drop
  - alterColumns
```
public void alterColumns(List<ColumnAlteration> columnAlterations)
```
    Alter columns in the dataset.
    
    Parameters:
    
    columnAlterations - The list of columns need to be altered.
  - newScan
```
public LanceScanner newScan()
```
    Create a new Dataset Scanner.
    
    Returns:
    
    a dataset scanner
  - newScan
```
public LanceScanner newScan(long batchSize)
```
    Create a new Dataset Scanner.
    
    Parameters:
    
    batchSize - the scan options with batch size, columns filter, and substrait
    
    Returns:
    
    a dataset scanner
  - newScan
```
public LanceScanner newScan(ScanOptions options)
```
    Create a new Dataset Scanner.
    
    Parameters:
    
    options - the scan options
    
    Returns:
    
    a dataset scanner
  - take
```
public org.apache.arrow.vector.ipc.ArrowReader take(List<Long> indices,
                                                    List<String> columns)
                                             throws IOException
```
    Select rows of data by index.
    
    Parameters:
    
    indices - the indices to take
    
    columns - the columns to take
    
    Returns:
    
    an ArrowReader
    
    Throws:
    
    IOException
  - delete
```
public void delete(String predicate)
```
    Delete rows of data by predicate.
    
    Parameters:
    
    predicate - the predicate to delete
  - uri
```
public String uri()
```
    Gets the URI of the dataset.
    
    Returns:
    
    the URI of the dataset
  - version
```
public long version()
```
    Get the currently checked out version id of the dataset
    
    Returns:
    
    the version id of the dataset
  - getVersion
```
public Version getVersion()
```
    Gets the currently checked out version of the dataset.
    
    Returns:
    
    the version of the dataset
  - listVersions
```
public List<Version> listVersions()
```
    Get the version history of the dataset.
    
    Returns:
    
    the version history of the dataset
  - latestVersion
```
public long latestVersion()
```
    Returns:
    
    the latest version of the dataset.
  - checkoutLatest
```
public void checkoutLatest()
```
    Checkout the dataset to the latest version.
  - checkoutVersion
```
public Dataset checkoutVersion(long version)
```
    Checks out a specific version of the dataset. If the version is already checked out, it returns a new Java Dataset object pointing to the same underlying Rust Dataset object
    
    Parameters:
    
    version - the version to check out
    
    Returns:
    
    a new Dataset instance with the specified version checked out
  - checkoutTag
```
public Dataset checkoutTag(String tag)
```
    Checks out a specific tag of the dataset. If the underlying version is already checked out, it returns a new Java Dataset object pointing to the same underlying Rust Dataset object
    
    Parameters:
    
    tag - the tag to check out
    
    Returns:
    
    a new Dataset instance with the specified tag checked out
  - restore
```
public void restore()
```
    Restore the currently checked out version of the dataset as the latest version. This operation produces a new version and doesn't influence any old versions and tags.
  - createIndex
```
public void createIndex(List<String> columns,
                        IndexType indexType,
                        Optional<String> name,
                        IndexParams params,
                        boolean replace)
```
    Creates a new index on the dataset. Only vector indexes are supported.
    
    Parameters:
    
    columns - the columns to index from
    
    indexType - the index type
    
    name - the name of the created index
    
    params - index params
    
    replace - whether to replace the existing index
  - countRows
```
public long countRows()
```
    Count the number of rows in the dataset.
    
    Returns:
    
    num of rows
  - countRows
```
public long countRows(String filter)
```
    Count the number of rows in the dataset.
    
    Parameters:
    
    filter - the filter expr to count row
    
    Returns:
    
    num of rows
  - calculateDataSize
```
public long calculateDataSize()
```
    Calculate the size of the dataset.
    
    Returns:
    
    the size of the dataset
  - getFragments
```
public List<Fragment> getFragments()
```
    Get all fragments in this dataset.
    
    Returns:
    
    A list of Fragment.
  - getSchema
```
public org.apache.arrow.vector.types.pojo.Schema getSchema()
```
    Gets the arrow schema of the dataset.
    
    Returns:
    
    the arrow schema
  - getLanceSchema
```
public LanceSchema getLanceSchema()
```
    Get the LanceSchema of the dataset with field ids.
    
    Returns:
    
    the LanceSchema
  - readTransaction
```
public Optional<Transaction> readTransaction()
```
    Get the Transaction of the dataset at the current version.
    
    Returns:
    
    the Transaction
  - listIndexes
```
public List<String> listIndexes()
```
    Returns:
    
    all the created indexes names
  - getConfig
```
public Map<String,String> getConfig()
```
    Get the table config of the dataset.
    
    Returns:
    
    the table config
  - compact
```
public void compact(CompactionOptions options)
```
    Compact the dataset to improve performance.
    This operation performs several optimizations:
    - Removes deleted rows from fragments
    - Removes dropped columns from fragments
    - Merges fragments that are too small
    Parameters:
    
    options - compaction options to control the behavior
  - compact
```
public void compact()
```
    Compact the dataset with default options.
  - updateConfig
```
@Deprecated
public void updateConfig(Map<String,String> tableConfig)
```
    Deprecated. Use newTransactionBuilder() with UpdateConfig operation instead
    
    Update the config of the dataset. This operation will only overwrite and NOT delete the existing config.
    
    Parameters:
    
    tableConfig - the config to update
  - deleteConfigKeys
```
@Deprecated
public void deleteConfigKeys(Set<String> deleteKeys)
```
    Deprecated. Use newTransactionBuilder() with UpdateConfig operation instead
    
    Delete the config keys of the dataset.
    
    Parameters:
    
    deleteKeys - the config keys to delete
  - close
```
public void close()
```
    Closes this dataset and releases any system resources associated with it. If the dataset is already closed, then invoking this method has no effect.
    
    Specified by:
    
    close in interface Closeable
    
    Specified by:
    
    close in interface AutoCloseable
  - takeBlobs
```
public List<BlobFile> takeBlobs(List<Long> rowIds,
                                String column)
```
    Open blob files for given row ids on a blob column. Names and semantics align with Rust/Python.
    
    Parameters:
    
    rowIds - stable row ids (row addresses)
    
    column - blob column name
    
    Returns:
    
    list of BlobFile objects
  - takeBlobsByIndices
```
public List<BlobFile> takeBlobsByIndices(List<Long> rowIndices,
                                         String column)
```
    Open blob files for given row indices on a blob column.
    
    Parameters:
    
    rowIndices - row offsets within dataset
    
    column - blob column name
    
    Returns:
    
    list of BlobFile objects
  - closed
```
public boolean closed()
```
    Checks if the dataset is closed.
    
    Returns:
    
    true if the dataset is closed, false otherwise.
  - getFragment
```
public Fragment getFragment(int fragmentId)
```
  - tags
```
public Dataset.Tags tags()
```
    Returns a Dataset.Tags instance for performing tag-related operations on the dataset.
    
    Returns:
    
    new Tags instance for dataset tag operations
    
    See Also:
    
    Dataset.Tags
  - branches
```
public Dataset.Branches branches()
```
    Branch operations aligned with Rust's Dataset branch APIs.
  - checkout
```
public Dataset checkout(Ref ref)
```
    Checkout using a unified Ref which can be a tag, the latest version on main/branch or a specified (branch_name, version_number).
    
    Parameters:
    
    ref - the checkout reference
    
    Returns:
    
    a new Dataset instance checked out to the specified reference
  - getTableMetadata
```
public Map<String,String> getTableMetadata()
```
    Get the table metadata of the dataset.
    
    Returns:
    
    the table metadata as a map of key-value pairs
  - sql
```
public SqlQuery sql(String sql)
```
    Execute SQL query on the dataset. The underlying SQL engine is DataFusion. Please refer to the DataFusion documentation for supported SQL syntax.
    
    Parameters:
    
    sql - SELECT statement to execute. The default FROM table name is `dataset`, for example: SELECT * FROM `dataset` LIMIT 10. If FROM table name is a custom value, the SqlQuery.tableName(String) should be invoked to set the custom table name.
    
    Returns:
    
    a SqlQuery instance.
  - mergeInsert
```
public MergeInsertResult mergeInsert(MergeInsertParams mergeInsert,
                                     org.apache.arrow.c.ArrowArrayStream source)
```
    Merge source data with the existing target data.
    This will take in the source, merge it with the existing target data, and insert new rows, update existing rows, and delete existing rows.
    It is important that after merge insert, the current dataset is changed and should be closed. The merged new dataset is contained in the MergeInsertResult.
    
    Parameters:
    
    mergeInsert - merge insert options
    
    source - ArrowArrayStream source data
    
    Returns:
    
    MergeInsertResult containing the new merged Dataset.
  - shallowClone
```
public Dataset shallowClone(String targetPath,
                            Ref ref)
```
  - shallowClone
```
public Dataset shallowClone(String targetPath,
                            Ref ref,
                            Map<String,String> storageOptions)
```
    Shallow clone the specified tag into a new dataset at the target path.
    This creates a new dataset that references the data files from the source dataset without copying them. Only metadata is written at the destination.
    
    Parameters:
    
    targetPath - the URI to clone the dataset into
    
    ref - the referred version of the current dataset
    
    storageOptions - Optional object store options for the destination dataset; empty uses default store parameters
    
    Returns:
    
    a new Dataset instance at the target path

Class Dataset

Nested Class Summary

Method Summary

Methods inherited from class java.lang.Object

Method Detail

create

create

open

open

open

open

commit

commit

commitAppend

commitOverwrite

allocator

newTransactionBuilder

commitTransaction

drop

addColumns

addColumns

addColumns

addColumns

dropColumns

alterColumns

newScan

newScan

newScan

take

delete

uri

version

getVersion

listVersions

latestVersion

checkoutLatest

checkoutVersion

checkoutTag

restore

createIndex

countRows

countRows

calculateDataSize

getFragments

getSchema

getLanceSchema

readTransaction

listIndexes

getConfig

compact

compact

updateConfig

deleteConfigKeys

close

takeBlobs

takeBlobsByIndices

closed

getFragment

tags

branches

checkout

getTableMetadata

sql

mergeInsert

shallowClone

shallowClone