com.lancedb.lance.io

Interface StorageOptionsProvider

All Known Implementing Classes:

LanceNamespaceStorageOptionsProvider

public interface StorageOptionsProvider

Interface for providing cloud storage options to Lance datasets.

Storage options providers enable automatic refresh for long-running operations on cloud storage (S3, Azure, GCS). This is currently only used for refreshing AWS temporary access credentials. Implement this interface to integrate with custom credential management systems such as AWS STS, GCP STS, or proprietary credential services.

The provider is called automatically before storage options expire, ensuring uninterrupted access during long-running queries, training jobs, or data processing.

Example Implementation

 public class MyStorageOptionsProvider implements StorageOptionsProvider {
   public Map<String, String> fetchStorageOptions() {
     // Fetch from your credential service
     Map<String, String> credentials = new HashMap<>();
     credentials.put("aws_access_key_id", "ASIA...");
     credentials.put("aws_secret_access_key", "secret");
     credentials.put("aws_session_token", "token");

     long expiresAtMillis = System.currentTimeMillis() + 3600000L;
     credentials.put("expires_at_millis", String.valueOf(expiresAtMillis));

     return credentials;
   }
 }

 // Use with dataset
 StorageOptionsProvider vendor = new MyStorageOptionsProvider();
 Dataset dataset = Dataset.open(
     "s3://bucket/table.lance",
     new ReadOptions.Builder()
         .setStorageOptionsProvider(vendor)
         .build()
 );
 

Error Handling

If fetchStorageOptions() throws an exception, operations requiring credentials will fail. Implementations should handle recoverable errors internally (e.g., retry token refresh) and only throw exceptions for unrecoverable errors.

Method Summary

Modifier and Type	Method and Description
Map<String,String>	fetchStorageOptions() Fetch fresh storage credentials.
default String	providerId() Return a human-readable unique identifier for this provider instance.

Method Detail

fetchStorageOptions

Map<String,String> fetchStorageOptions()

Fetch fresh storage credentials.

This method is called automatically before each request and before existing credentials expire. It must return credentials in the format described below.

Returns:

Map of string key-value pairs containing cloud storage credentials and expiration time. Required key:

• "expires_at_millis" (String): Unix timestamp in milliseconds (as string) when credentials expire. Lance will automatically call fetchStorageOptions() again before this time.

Plus provider-specific credential keys:

• AWS S3: "aws_access_key_id", "aws_secret_access_key", "aws_session_token" (optional)
• Azure Blob Storage: "account_name", "account_key" or "sas_token"
• Google Cloud Storage: "service_account_key" or "token"

Throws:

RuntimeException - if unable to fetch credentials

providerId

default String providerId()

Return a human-readable unique identifier for this provider instance.

This is used for equality comparison and hashing in the object store registry. Two providers with the same ID will be treated as equal and share the same cached ObjectStore instance.

The default implementation uses the class name and toString() representation. Override this method to provide semantic equality based on configuration.

Returns:

A human-readable unique identifier string. For example: "MyProvider { endpoint: 'https://api.example.com' }"