Package io.delta.kernel.internal.replay

Class LogReplay

Object
io.delta.kernel.internal.replay.LogReplay
public class LogReplay extends Object
Replays a history of actions, resolving them to produce the current state of the table. The protocol for resolution is as follows:
  • The most recent AddFile and accompanying metadata for any `(path, dv id)` tuple wins.
  • RemoveFile deletes a corresponding AddFile. A RemoveFile "corresponds" to the AddFile that matches both the parquet file URI *and* the deletion vector's URI (if any).
  • The most recent Metadata wins.
  • The most recent Protocol version wins.
  • For each `(path, dv id)` tuple, this class should always output only one * FileAction (either AddFile or RemoveFile)

  • Field Details

    • SET_TRANSACTION_READ_SCHEMA

      public static final StructType SET_TRANSACTION_READ_SCHEMA
      Read schema when searching for just the transaction identifiers

    • DOMAIN_METADATA_READ_SCHEMA

      public static final StructType DOMAIN_METADATA_READ_SCHEMA
      Read schema when searching for just the domain metadata

    • SIDECAR_FIELD_NAME

      public static String SIDECAR_FIELD_NAME

    • ADDFILE_FIELD_NAME

      public static String ADDFILE_FIELD_NAME

    • REMOVEFILE_FIELD_NAME

      public static String REMOVEFILE_FIELD_NAME

    • ADD_FILE_ORDINAL

      public static int ADD_FILE_ORDINAL

    • ADD_FILE_PATH_ORDINAL

      public static int ADD_FILE_PATH_ORDINAL

    • ADD_FILE_DV_ORDINAL

      public static int ADD_FILE_DV_ORDINAL

    • REMOVE_FILE_ORDINAL

      public static int REMOVE_FILE_ORDINAL

    • REMOVE_FILE_PATH_ORDINAL

      public static int REMOVE_FILE_PATH_ORDINAL

    • REMOVE_FILE_DV_ORDINAL

      public static int REMOVE_FILE_DV_ORDINAL

  • Constructor Details

    • LogReplay

      public LogReplay(Engine engine, Path dataPath, Lazy<LogSegment> lazyLogSegment, Lazy<Optional<CRCInfo>> lazyCrcInfo)
      Creates a new LogReplay instance.
      Parameters:
      engine - the engine to use for reading log files
      dataPath - the path to the Delta table
      lazyLogSegment - lazy loader for the log segment
      lazyCrcInfo - lazy loader for the CRC file (shared with ProtocolMetadataLogReplay)

  • Method Details

    • withSidecarFileSchema

      public static StructType withSidecarFileSchema(StructType schema)

    • containsAddOrRemoveFileActions

      public static boolean containsAddOrRemoveFileActions(StructType schema)

    • getAddRemoveReadSchema

      public static StructType getAddRemoveReadSchema(boolean shouldReadStats)
      Read schema when searching for all the active AddFiles

    • getAddReadSchema

      public static StructType getAddReadSchema(boolean shouldReadStats)
      Read schema when searching only for AddFiles

    • getVersion

      public long getVersion()

    • getLatestTransactionIdentifier

      public Optional<Long> getLatestTransactionIdentifier(Engine engine, String applicationId)

    • getActiveDomainMetadataMap

      public Map<String,DomainMetadata> getActiveDomainMetadataMap()
      Returns map for all active domain metadata.

    • getCrcInfoAtSnapshotVersion

      public Optional<CRCInfo> getCrcInfoAtSnapshotVersion()
      Returns the CRC info for the current snapshot version if available. Lazily loads and caches the CRC file on first access. Returns empty if no CRC file exists at the snapshot version.

    • getAddFilesAsColumnarBatches

      public CloseableIterator<FilteredColumnarBatch> getAddFilesAsColumnarBatches(Engine engine, boolean shouldReadStats, Optional<Predicate> checkpointPredicate, ScanMetrics scanMetrics, Optional<PaginationContext> paginationContextOpt)
      Returns an iterator of FilteredColumnarBatch representing all the active AddFiles in the table.

      Statistics are conditionally read for the AddFiles based on shouldReadStats. The returned batches have schema:

      1. name: add

        type: AddFile.SCHEMA_WITH_STATS if shouldReadStats=true, otherwise AddFile.SCHEMA_WITHOUT_STATS

    • getLogSegment

      public LogSegment getLogSegment()