Package io.delta.kernel.internal

Class DeltaLogActionUtils

Object

io.delta.kernel.internal.DeltaLogActionUtils

public class DeltaLogActionUtils
extends Object

Exposes APIs to read the raw actions within the *commit files* of the _delta_log. This is used for CDF, streaming, and more.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static enum	DeltaLogActionUtils.DeltaAction	Represents a Delta action.

Method Summary

Modifier and Type	Method	Description
static CloseableIterator<CommitActions>	getActionsFromCommitFilesWithProtocolValidation(Engine engine, String tablePath, List<FileStatus> commitFiles, Set<DeltaLogActionUtils.DeltaAction> actionSet)	Returns CommitActions for each commit file.
static List<FileStatus>	getCommitFilesForVersionRange(Engine engine, Path tablePath, long startVersion, Optional<Long> endVersionOpt)	For a table get the list of commit log files for the provided version range.
static CloseableIterator<FileStatus>	listDeltaLogFilesAsIter(Engine engine, Set<FileNames.DeltaLogFileType> fileTypes, Path tablePath, long startVersion, Optional<Long> endVersionOpt, boolean mustBeRecreatable)	Returns a CloseableIterator of files of type $fileTypes in the _delta_log directory of the given $tablePath, in increasing order from $startVersion to the optional $endVersion.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

getCommitFilesForVersionRange

public static List<FileStatus> getCommitFilesForVersionRange(Engine engine,
 Path tablePath,
 long startVersion,
 Optional<Long> endVersionOpt)

For a table get the list of commit log files for the provided version range.

Parameters:

tablePath - path for the given table

startVersion - start version of the range (inclusive)

endVersionOpt - end version of the range (inclusive)

Returns:

the list of commit files in increasing order between startVersion and endVersion

Throws:

TableNotFoundException - if the table does not exist or if it is not a delta table

KernelException - if a commit file does not exist for any of the versions in the provided range

KernelException - if provided an invalid version range

listDeltaLogFilesAsIter

public static CloseableIterator<FileStatus> listDeltaLogFilesAsIter(Engine engine,
 Set<FileNames.DeltaLogFileType> fileTypes,
 Path tablePath,
 long startVersion,
 Optional<Long> endVersionOpt,
 boolean mustBeRecreatable)

Returns a CloseableIterator of files of type $fileTypes in the _delta_log directory of the given $tablePath, in increasing order from $startVersion to the optional $endVersion.

Throws:

TableNotFoundException - if the table or its _delta_log does not exist

KernelException - if mustBeRecreatable is true, endVersionOpt is present, and the _delta_log history has been truncated so that we cannot load the desired end version

getActionsFromCommitFilesWithProtocolValidation

public static CloseableIterator<CommitActions> getActionsFromCommitFilesWithProtocolValidation(Engine engine,
 String tablePath,
 List<FileStatus> commitFiles,
 Set<DeltaLogActionUtils.DeltaAction> actionSet)

Returns CommitActions for each commit file. CommitActions are ordered by increasing version.

This function automatically:

• Performs protocol validation by reading and validating the protocol action
• Extracts commit timestamp using inCommitTimestamp if available, otherwise file modification time
• Filters out protocol and commitInfo actions if not requested in actionSet