Package io.delta.kernel.internal.snapshot

Class LogSegment

Object
io.delta.kernel.internal.snapshot.LogSegment
public class LogSegment extends Object

  • Constructor Details

    • LogSegment

      public LogSegment(Path logPath, long version, List<FileStatus> deltas, List<FileStatus> compactions, List<FileStatus> checkpoints, FileStatus deltaAtEndVersion, Optional<FileStatus> lastSeenChecksum, Optional<Long> maxPublishedDeltaVersion)
      Provides information around which files in the transaction log need to be read to create the given version of the log.

      This constructor validates and guarantees that:

      • All deltas are valid deltas files
      • All checkpoints are valid checkpoint files
      • All checkpoint files have the same version
      • All deltas are contiguous and range from checkpointVersionOpt + 1 to version
      • If no deltas are present then checkpointVersionOpt is equal to version
      Parameters:
      logPath - The path to the _delta_log directory
      version - The Snapshot version to generate
      deltas - The delta commit files (.json) to read
      compactions - Any found log compactions files that can be used in place of some or all of the deltas
      checkpoints - The checkpoint file(s) to read
      deltaAtEndVersion - The delta file at the end version of this LogSegment. If this LogSegment contains only checkpoints (e.g. 10.checkpoint only) then this is the delta at that checkpoint version.
      lastSeenChecksum - The most recent checksum file encountered during log directory listing, if available.
      maxPublishedDeltaVersion - The maximum version among all published delta files seen during log segment construction, if available. Note that the Published Delta file for this version may not be included as a Delta in this LogSegment, if there was a catalog commit that took priority over it.

  • Method Details

    • createForNewTable

      public static LogSegment createForNewTable(Path logPath, ParsedDeltaData parsedDeltaVersion0)
      Creates a LogSegment for a newly created table from a single ParsedDeltaData. Used to construct a post-commit Snapshot after a CREATE transaction.
      Parameters:
      logPath - The path to the _delta_log directory
      parsedDeltaVersion0 - The ParsedDeltaData that must be for version 0
      Returns:
      A new LogSegment with just this delta
      Throws:
      IllegalArgumentException - if the ParsedDeltaData is not file-based

    • getLogPath

      public Path getLogPath()

    • getVersion

      public long getVersion()

    • getDeltas

      public List<FileStatus> getDeltas()

    • getCompactions

      public List<FileStatus> getCompactions()

    • getCheckpoints

      public List<FileStatus> getCheckpoints()

    • getCheckpointVersionOpt

      public Optional<Long> getCheckpointVersionOpt()

    • getLastSeenChecksum

      public Optional<FileStatus> getLastSeenChecksum()
      Returns the most recent checksum file encountered during log directory listing, if available.

      Note: This checksum file's version is guaranteed to:

      • Be less than or equal to the LogSegment version (enforced by constructor)
      • Be greater than or equal to the checkpoint version if a checkpoint exists (filtered during initialization)
      Returns:
      Optional containing the most recent valid checksum file encountered, or empty if none found

    • getMaxPublishedDeltaVersion

      public Optional<Long> getMaxPublishedDeltaVersion()
      Returns the maximum published delta version observed during log segment construction.

      This is a best-effort API that returns what was actually seen during construction, not the authoritative maximum published delta version in the log.

      Optional.empty() means "we don't know" - not necessarily that no deltas have been published. This can occur when:

      • Only checkpoint files were found during listing (e.g., due to log cleanup)
      • Listing bounds did not include published delta files
      • The table contains only catalog commits with no published deltas
      Returns:
      the maximum published delta version seen during construction, or empty if unknown

    • getDeltaFileAtEndVersion

      public FileStatus getDeltaFileAtEndVersion()
      Returns the Delta file at the end version of this LogSegment.

      If this LogSegment has checkpoints and deltas, then this is the last delta.

      If this LogSegment has only checkpoints (i.e. 10.checkpoint only) then this is the delta at that checkpoint version.

    • allLogFilesReversed

      public List<FileStatus> allLogFilesReversed()
      Returns:
      all deltas (.json) and checkpoint (.checkpoint.parquet) files in this LogSegment, sorted in reverse (00012.json, 00011.json, 00010.checkpoint.parquet) order.

    • allFilesWithCompactionsReversed

      public List<FileStatus> allFilesWithCompactionsReversed()
      Returns:
      all files sorted in reverse order in this log segment, but omitting the deltas (.json) files that are covered by log compaction files. This will include deltas (xxx.json) that are not covered by a log compaction, compaction files (xxx.xxx.json), and checkpoints (.checkpoint.parquet).

    • getAllCatalogCommits

      public List<ParsedCatalogCommitData> getAllCatalogCommits()

    • newWithAddedDeltas

      public LogSegment newWithAddedDeltas(List<ParsedDeltaData> addedDeltas)
      Creates a new LogSegment by extending this LogSegment with additional deltas. Used to construct a post-commit Snapshot from a previous Snapshot.

      The additional deltas must be contiguous and start at version + 1.

      Parameters:
      addedDeltas - List of ParsedDeltaData to add (must be contiguous and start at current version + 1)
      Returns:
      A new LogSegment with the additional deltas
      Throws:
      IllegalArgumentException - if deltas are not contiguous or don't start at version + 1

    • newAsPublished

      public LogSegment newAsPublished()
      Creates a new LogSegment that reflects the published commits. Used to construct a post-publish Snapshot from a previous Snapshot.
      Returns:
      A new LogSegment with published commits

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Object