Package io.delta.kernel.internal.replay

Class ActionsIterator

Object
io.delta.kernel.internal.replay.ActionsIterator
All Implemented Interfaces:
CloseableIterator<ActionWrapper>, Closeable, AutoCloseable, Iterator<ActionWrapper>
public class ActionsIterator extends Object implements CloseableIterator<ActionWrapper>
This class takes as input a list of delta files (.json, .checkpoint.parquet) and produces an iterator of (ColumnarBatch, isFromCheckpoint) tuples, where the schema of the ColumnarBatch semantically represents actions (or, a subset of action fields) parsed from the Delta Log.

Users must pass in a `deltaReadSchema` to select which actions and sub-fields they want to consume.

Users can also pass in an optional `checkpointReadSchema` if it is different from `deltaReadSchema`.

  • Constructor Details

  • Method Details

    • paginatedFilter

      @VisibleForTesting public boolean paginatedFilter(DeltaLogFile nextLogFile)
      Filters a log segment file based on the pagination context.

      If this method returns true, the current file will be kept; otherwise, it will be skipped.

      • If pagination is not enabled (i.e., paginationContextOpt is not present), return true.
      • If the pagination context is present but doesn't include a last read log file path, return true (indicates reading the first page).
      • If the file is a JSON log file, return true — we never skip JSON files as they're needed to build hash sets.
      • If the file is a V2 checkpoint manifest, return true — these should never be skipped.
      • If the file is a checkpoint file and comes after the last log file recorded in the page token, return false (skip it).

      Note: The nextLogFile parameter cannot be a sidecar file because sidecar files are not included in the log segment list. Sidecar files are handled separately later, after the V2 manifest file has been read, specifically in the extractSidecarFiles() method.

      Parameters:
      nextLogFile - the log file to evaluate
      Returns:
      true to include the file; false to skip it

    • hasNext

      public boolean hasNext()
      Description copied from interface: CloseableIterator
      Returns true if the iteration has more elements. (In other words, returns true if next would return an element rather than throwing an exception.)
      Specified by:
      hasNext in interface CloseableIterator<ActionWrapper>
      Specified by:
      hasNext in interface Iterator<ActionWrapper>
      Returns:
      true if the iteration has more elements

    • next

      public ActionWrapper next()
      Description copied from interface: CloseableIterator
      Returns the next element in the iteration.
      Specified by:
      next in interface CloseableIterator<ActionWrapper>
      Specified by:
      next in interface Iterator<ActionWrapper>
      Returns:
      a tuple of (ColumnarBatch, isFromCheckpoint), where ColumnarBatch conforms to the instance deltaReadSchema or checkpointReadSchema (the latter when when isFromCheckpoint=true).

    • close

      public void close() throws IOException
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException

    • extractSidecarsFromBatch

      public ColumnarBatch extractSidecarsFromBatch(FileStatus checkpointFileStatus, long checkpointVersion, ColumnarBatch columnarBatch)
      Reads SidecarFile actions from ColumnarBatch, removing sidecar actions from the ColumnarBatch. Returns a list of SidecarFile actions found.