Index

A B C D E F G H I L N P R S T U W
All Classes and Interfaces|All Packages|Constant Field Values|Serialized Form

A

AbstractCommitInfo - Interface in io.delta.storage.commit.actions

Interface for objects that represents the base information for a commit.

AbstractMetadata - Interface in io.delta.storage.commit.actions

Interface for metadata actions in Delta.

AbstractProtocol - Interface in io.delta.storage.commit.actions

Interface for protocol actions in Delta.

acquire(Path) - Method in class io.delta.storage.internal.PathLock

Acquire a lock for the path before writing.

AzureLogStore - Class in io.delta.storage

LogStore implementation for Azure.

AzureLogStore(Configuration) - Constructor for class io.delta.storage.AzureLogStore

B

backfillToVersion(LogStore, Configuration, TableDescriptor, long, Long) - Method in interface io.delta.storage.commit.CommitCoordinatorClient

API to ask the commit coordinator client to backfill all commits up to version and notify the commit coordinator.

C

close() - Method in class io.delta.storage.LineCloseableIterator

CloseableIterator<T> - Interface in io.delta.storage

:: DeveloperApi :: An iterator that may contain resources which should be released after use.

commit(LogStore, Configuration, TableDescriptor, long, Iterator<String>, UpdatedActions) - Method in interface io.delta.storage.commit.CommitCoordinatorClient

API to commit the given set of actions to the table represented by logPath at the given commitVersion.

Commit - Class in io.delta.storage.commit

Representation of a commit file

Commit(long, FileStatus, long) - Constructor for class io.delta.storage.commit.Commit

CommitCoordinatorClient - Interface in io.delta.storage.commit

The CommitCoordinatorClient is responsible for communicating with the commit coordinator and backfilling commits.

commitDirPath(Path) - Static method in class io.delta.storage.commit.CoordinatedCommitsUtils

Returns path to the directory which holds the unbackfilled commits

CommitFailedException - Exception in io.delta.storage.commit

Exception raised by CommitCoordinatorClient.commit(LogStore, Configuration, TableDescriptor, long, Iterator, UpdatedActions)

CommitFailedException(boolean, boolean, String) - Constructor for exception io.delta.storage.commit.CommitFailedException

CommitFailedException(boolean, boolean, String, Throwable) - Constructor for exception io.delta.storage.commit.CommitFailedException

CommitResponse - Class in io.delta.storage.commit

Response container for CommitCoordinatorClient.commit(LogStore, Configuration, TableDescriptor, long, Iterator, UpdatedActions).

CommitResponse(Commit) - Constructor for class io.delta.storage.commit.CommitResponse

CoordinatedCommitsUtils - Class in io.delta.storage.commit

D

deltaFile(Path, long) - Static method in class io.delta.storage.internal.FileNameUtils

Returns the delta (json format) path for a given delta file.

deltaVersion(Path) - Static method in class io.delta.storage.internal.FileNameUtils

Returns the version for the given delta path.

E

equals(Object) - Method in class io.delta.storage.commit.Commit

F

FileNameUtils - Class in io.delta.storage.internal

Helper for misc functions relating to file names for delta commits.

FileNameUtils() - Constructor for class io.delta.storage.internal.FileNameUtils

G

GCSLogStore - Class in io.delta.storage

The LogStore implementation for GCS, which uses gcs-connector to provide the necessary atomic and durability guarantees: Atomic Visibility: Read/read-after-metadata-update/delete are strongly consistent for GCS. Consistent Listing: GCS guarantees strong consistency for both object and bucket listing operations.

GCSLogStore(Configuration) - Constructor for class io.delta.storage.GCSLogStore

generateUnbackfilledDeltaFilePath(Path, long) - Static method in class io.delta.storage.commit.CoordinatedCommitsUtils

Creates a new unbackfilled delta file path for the given commit version.

getBackfilledDeltaFilePath(Path, Long) - Static method in class io.delta.storage.commit.CoordinatedCommitsUtils

Returns the path to the backfilled delta file for the given commit version.

getCommit() - Method in class io.delta.storage.commit.CommitResponse

getCommitInfo() - Method in class io.delta.storage.commit.UpdatedActions

getCommits() - Method in class io.delta.storage.commit.GetCommitsResponse

getCommits(TableDescriptor, Long, Long) - Method in interface io.delta.storage.commit.CommitCoordinatorClient

API to get the unbackfilled commits for the table represented by the given logPath.

GetCommitsResponse - Class in io.delta.storage.commit

Response container for CommitCoordinatorClient.getCommits(TableDescriptor, Long, Long).

GetCommitsResponse(List<Commit>, long) - Constructor for class io.delta.storage.commit.GetCommitsResponse

getCommitTimestamp() - Method in interface io.delta.storage.commit.actions.AbstractCommitInfo

Get the timestamp of the commit as millis after the epoch.

getCommitTimestamp() - Method in class io.delta.storage.commit.Commit

getConfiguration() - Method in interface io.delta.storage.commit.actions.AbstractMetadata

The table properties defined on the table.

getConflict() - Method in exception io.delta.storage.commit.CommitFailedException

getCoordinatorConf(AbstractMetadata) - Static method in class io.delta.storage.commit.CoordinatedCommitsUtils

Get the coordinated commits owner configuration from the provided abstract metadata.

getCoordinatorName(AbstractMetadata) - Static method in class io.delta.storage.commit.CoordinatedCommitsUtils

Retrieves the coordinator name from the provided abstract metadata.

getCreatedTime() - Method in interface io.delta.storage.commit.actions.AbstractMetadata

Timestamp for the creation of this metadata.

getDescription() - Method in interface io.delta.storage.commit.actions.AbstractMetadata

User-specified table description.

getFileStatus() - Method in class io.delta.storage.commit.Commit

getFormatOptions() - Method in interface io.delta.storage.commit.actions.AbstractMetadata

The format options

getId() - Method in interface io.delta.storage.commit.actions.AbstractMetadata

A unique table identifier.

getLatestTableVersion() - Method in class io.delta.storage.commit.GetCommitsResponse

getLogPath() - Method in class io.delta.storage.commit.TableDescriptor

getMinReaderVersion() - Method in interface io.delta.storage.commit.actions.AbstractProtocol

The minimum reader version required to read the table.

getMinWriterVersion() - Method in interface io.delta.storage.commit.actions.AbstractProtocol

The minimum writer version required to read the table.

getName() - Method in interface io.delta.storage.commit.actions.AbstractMetadata

User-specified table identifier.

getName() - Method in class io.delta.storage.commit.TableIdentifier

Returns the name of the table.

getNamespace() - Method in class io.delta.storage.commit.TableIdentifier

Returns the namespace of the table.

getNewMetadata() - Method in class io.delta.storage.commit.UpdatedActions

getNewProtocol() - Method in class io.delta.storage.commit.UpdatedActions

getOldMetadata() - Method in class io.delta.storage.commit.UpdatedActions

getOldProtocol() - Method in class io.delta.storage.commit.UpdatedActions

getPartitionColumns() - Method in interface io.delta.storage.commit.actions.AbstractMetadata

List of partition columns.

getProvider() - Method in interface io.delta.storage.commit.actions.AbstractMetadata

The table provider format.

getReaderFeatures() - Method in interface io.delta.storage.commit.actions.AbstractProtocol

The reader features that need to be supported to read the table.

getRetryable() - Method in exception io.delta.storage.commit.CommitFailedException

getSchemaString() - Method in interface io.delta.storage.commit.actions.AbstractMetadata

The table schema in string representation.

getTableConf() - Method in class io.delta.storage.commit.TableDescriptor

getTableConf(AbstractMetadata) - Static method in class io.delta.storage.commit.CoordinatedCommitsUtils

Get the coordinated commits table configuration from the provided abstract metadata.

getTableIdentifier() - Method in class io.delta.storage.commit.TableDescriptor

getTablePath(Path) - Static method in class io.delta.storage.commit.CoordinatedCommitsUtils

Get the table path from the provided log path.

getUnbackfilledDeltaFile(Path, long, Optional<String>) - Static method in class io.delta.storage.commit.CoordinatedCommitsUtils

Returns the un-backfilled uuid formatted delta (json format) path for a given version.

getVersion() - Method in class io.delta.storage.commit.Commit

getWriterFeatures() - Method in interface io.delta.storage.commit.actions.AbstractProtocol

The writer features that need to be supported to write the table.

H

HadoopFileSystemLogStore - Class in io.delta.storage

Default implementation of LogStore for Hadoop FileSystem implementations.

HadoopFileSystemLogStore(Configuration) - Constructor for class io.delta.storage.HadoopFileSystemLogStore

hashCode() - Method in class io.delta.storage.commit.Commit

hasNext() - Method in class io.delta.storage.LineCloseableIterator

HDFSLogStore - Class in io.delta.storage

The LogStore implementation for HDFS, which uses Hadoop FileContext API's to provide the necessary atomic and durability guarantees: Atomic visibility of files: `FileContext.rename` is used write files which is atomic for HDFS. Consistent file listing: HDFS file listing is consistent.

HDFSLogStore(Configuration) - Constructor for class io.delta.storage.HDFSLogStore

I

incorrectLogStoreImplementationException(Throwable) - Static method in class io.delta.storage.internal.LogStoreErrors

initHadoopConf() - Method in class io.delta.storage.LogStore

:: DeveloperApi :: Hadoop configuration that should only be used during initialization of LogStore.

io.delta.storage - package io.delta.storage

io.delta.storage.commit - package io.delta.storage.commit

io.delta.storage.commit.actions - package io.delta.storage.commit.actions

io.delta.storage.internal - package io.delta.storage.internal

isCoordinatedCommitsToFSConversion(Long, UpdatedActions) - Static method in class io.delta.storage.commit.CoordinatedCommitsUtils

Returns true if the commit is a coordinated commits to filesystem conversion.

isDeltaFile(Path) - Static method in class io.delta.storage.internal.FileNameUtils

Returns true if the given path is a delta file, else false.

isFatal(Throwable) - Static method in class io.delta.storage.internal.LogStoreErrors

Returns true if the provided Throwable is to be considered fatal, or false if it is to be considered non-fatal

isNonFatal(Throwable) - Static method in class io.delta.storage.internal.LogStoreErrors

Returns true if the provided Throwable is to be considered non-fatal, or false if it is to be considered fatal

isPartialWriteVisible(Path, Configuration) - Method in class io.delta.storage.AzureLogStore

isPartialWriteVisible(Path, Configuration) - Method in class io.delta.storage.GCSLogStore

isPartialWriteVisible(Path, Configuration) - Method in class io.delta.storage.HDFSLogStore

isPartialWriteVisible(Path, Configuration) - Method in class io.delta.storage.LocalLogStore

isPartialWriteVisible(Path, Configuration) - Method in class io.delta.storage.LogStore

:: DeveloperApi :: Whether a partial write is visible for the underlying file system of `path`.

isPartialWriteVisible(Path, Configuration) - Method in class io.delta.storage.S3SingleDriverLogStore

L

LineCloseableIterator - Class in io.delta.storage

Turn a Reader to CloseableIterator which can be read on demand.

LineCloseableIterator(Reader) - Constructor for class io.delta.storage.LineCloseableIterator

listFrom(Path, Configuration) - Method in class io.delta.storage.HadoopFileSystemLogStore

listFrom(Path, Configuration) - Method in class io.delta.storage.LogStore

:: DeveloperApi :: List the paths in the same directory that are lexicographically greater or equal to (UTF-8 sorting) the given `path`.

listFrom(Path, Configuration) - Method in class io.delta.storage.S3SingleDriverLogStore

LocalLogStore - Class in io.delta.storage

Default LogStore implementation (should be used for testing only!).

LocalLogStore(Configuration) - Constructor for class io.delta.storage.LocalLogStore

logDirPath(Path) - Static method in class io.delta.storage.commit.CoordinatedCommitsUtils

Returns path to the directory which holds the delta log

LogStore - Class in io.delta.storage

:: DeveloperApi ::

LogStore(Configuration) - Constructor for class io.delta.storage.LogStore

LogStoreErrors - Class in io.delta.storage.internal

LogStoreErrors() - Constructor for class io.delta.storage.internal.LogStoreErrors

N

next() - Method in class io.delta.storage.LineCloseableIterator

NO_ABSTRACT_FILE_SYSTEM_EXCEPTION_MESSAGE - Static variable in class io.delta.storage.HDFSLogStore

P

PathLock - Class in io.delta.storage.internal

A lock that provides per-file-path `acquire` and `release` semantics.

PathLock() - Constructor for class io.delta.storage.internal.PathLock

R

read(Path, Configuration) - Method in class io.delta.storage.HadoopFileSystemLogStore

read(Path, Configuration) - Method in class io.delta.storage.LogStore

:: DeveloperApi :: Load the given file and return an `Iterator` of lines, with line breaks removed from each line.

registerTable(Path, Optional<TableIdentifier>, long, AbstractMetadata, AbstractProtocol) - Method in interface io.delta.storage.commit.CommitCoordinatorClient

API to register the table represented by the given `logPath` at the provided currentTableVersion with the commit coordinator this commit coordinator client represents.

release(Path) - Method in class io.delta.storage.internal.PathLock

Release the lock for the path after writing.

resolvePathOnPhysicalStorage(Path, Configuration) - Method in class io.delta.storage.HadoopFileSystemLogStore

resolvePathOnPhysicalStorage(Path, Configuration) - Method in class io.delta.storage.LogStore

:: DeveloperApi :: Resolve the fully qualified path for the given `path`.

runInNewThread(String, boolean, Callable<T>) - Static method in class io.delta.storage.internal.ThreadUtils

Based on Apache Spark's ThreadUtils.runInNewThread Run a piece of code in a new thread and return the result.

S

s3ListFromArray(FileSystem, Path, Path) - Static method in class io.delta.storage.internal.S3LogStoreUtil

Uses the S3ListRequest.v2 interface with the startAfter parameter to only list files which are lexicographically greater than resolvedPath.

S3LogStoreUtil - Class in io.delta.storage.internal

Static utility methods for the S3SingleDriverLogStore.

S3SingleDriverLogStore - Class in io.delta.storage

Single JVM LogStore implementation for S3.

S3SingleDriverLogStore(Configuration) - Constructor for class io.delta.storage.S3SingleDriverLogStore

semanticEquals(CommitCoordinatorClient) - Method in interface io.delta.storage.commit.CommitCoordinatorClient

Determines whether this CommitCoordinatorClient is semantically equal to another CommitCoordinatorClient.

T

TableDescriptor - Class in io.delta.storage.commit

Container for all the info to uniquely identify the table

TableDescriptor(Path, Optional<TableIdentifier>, Map<String, String>) - Constructor for class io.delta.storage.commit.TableDescriptor

TableIdentifier - Class in io.delta.storage.commit

Identifier for a table.

TableIdentifier(String[], String) - Constructor for class io.delta.storage.commit.TableIdentifier

TableIdentifier(String, String...) - Constructor for class io.delta.storage.commit.TableIdentifier

ThreadUtils - Class in io.delta.storage.internal

ThreadUtils() - Constructor for class io.delta.storage.internal.ThreadUtils

U

UpdatedActions - Class in io.delta.storage.commit

A container class to inform the CommitCoordinatorClient about any changes in Protocol/Metadata

UpdatedActions(AbstractCommitInfo, AbstractMetadata, AbstractProtocol, AbstractMetadata, AbstractProtocol) - Constructor for class io.delta.storage.commit.UpdatedActions

W

withCommitTimestamp(long) - Method in class io.delta.storage.commit.Commit

withFileStatus(FileStatus) - Method in class io.delta.storage.commit.Commit

write(Path, Iterator<String>, Boolean, Configuration) - Method in class io.delta.storage.AzureLogStore

write(Path, Iterator<String>, Boolean, Configuration) - Method in class io.delta.storage.GCSLogStore

write(Path, Iterator<String>, Boolean, Configuration) - Method in class io.delta.storage.HDFSLogStore

write(Path, Iterator<String>, Boolean, Configuration) - Method in class io.delta.storage.LocalLogStore

This write implementation needs to wrap `writeWithRename` with `synchronized` as rename() for RawLocalFileSystem doesn't throw an exception when the target file exists.

write(Path, Iterator<String>, Boolean, Configuration) - Method in class io.delta.storage.LogStore

:: DeveloperApi :: Write the given `actions` to the given `path` with or without overwrite as indicated.

write(Path, Iterator<String>, Boolean, Configuration) - Method in class io.delta.storage.S3SingleDriverLogStore

writeUnbackfilledCommitFile(LogStore, Configuration, String, long, Iterator<String>, String) - Static method in class io.delta.storage.commit.CoordinatedCommitsUtils

Write a UUID-based commit file for the specified version to the table at logPath.

A B C D E F G H I L N P R S T U W
All Classes and Interfaces|All Packages|Constant Field Values|Serialized Form