Package io.delta.kernel.internal.actions

Class CommitInfo

Object
io.delta.kernel.internal.actions.CommitInfo
public class CommitInfo extends Object
Delta log action representing a commit information action. According to the Delta protocol there isn't any specific schema for this action, but we use the following schema:
  • inCommitTimestamp: Long - A monotonically increasing timestamp that represents the time since epoch in milliseconds when the commit write was started
  • timestamp: Long - Milliseconds since epoch UTC of when this commit happened
  • engineInfo: String - Engine that made this commit
  • operation: String - Operation (e.g. insert, delete, merge etc.)
  • operationParameters: Map(String, String) - each operation depending upon the type may add zero or more parameters about the operation. E.g. when creating a table `partitionBy` key with list of partition columns is added.
  • isBlindAppend: Boolean - Is this commit a blind append?
  • txnId: String - a unique transaction id of this commit
The Delta-Spark connector adds lot more fields to this action. We can add them as needed.

  • Field Details

    • FULL_SCHEMA

      public static final StructType FULL_SCHEMA

  • Constructor Details

  • Method Details

    • fromColumnVector

      public static CommitInfo fromColumnVector(ColumnVector vector, int rowId)

    • unsafeGetRequiredIctFromPublishedDeltaFile

      public static long unsafeGetRequiredIctFromPublishedDeltaFile(Engine engine, Path logPath, long version)
      Returns the `inCommitTimestamp` of delta file at the requested version. Throws an exception if the delta file does not exist or does not have a commitInfo action or if the commitInfo action contains an empty `inCommitTimestamp`.

      WARNING: UNSAFE METHOD because this assumes that 00N.json is published.

    • getRequiredIctFromDeltaFile

      public static long getRequiredIctFromDeltaFile(Engine engine, Path tablePath, FileStatus deltaFileStatus, long version)
      Returns the `inCommitTimestamp` of the provided delta file. Throws an exception if the delta file does not exist or does not have a commitInfo action or if the commitInfo action contains an empty `inCommitTimestamp`. The delta file can be either a published or staged commit file.

    • extractRequiredIctFromCommitInfoOpt

      public static long extractRequiredIctFromCommitInfoOpt(Optional<CommitInfo> commitInfoOpt, long version, Path dataPath)
      Returns the `inCommitTimestamp` of the given `commitInfoOpt` if it is defined. Throws an exception if `commitInfoOpt` is empty or contains an empty `inCommitTimestamp`.

    • unsafeTryReadCommitInfoFromPublishedDeltaFile

      public static Optional<CommitInfo> unsafeTryReadCommitInfoFromPublishedDeltaFile(Engine engine, Path logPath, long version)
      Get the CommitInfo action (if available) from the delta file at the given logPath and version.

      WARNING: UNSAFE METHOD because this assumes that 00N.json is published.

    • tryReadCommitInfoFromDeltaFile

      public static Optional<CommitInfo> tryReadCommitInfoFromDeltaFile(Engine engine, FileStatus deltaFileStatus)
      Read the CommitInfo action (if available) from the given delta file.

    • getTimestamp

      public long getTimestamp()

    • getEngineInfo

      public Optional<String> getEngineInfo()

    • getOperation

      public Optional<String> getOperation()

    • getOperationParameters

      public Map<String,String> getOperationParameters()

    • getIsBlindAppend

      public Optional<Boolean> getIsBlindAppend()

    • getTxnId

      public Optional<String> getTxnId()

    • getInCommitTimestamp

      public Optional<Long> getInCommitTimestamp()

    • getOperationMetrics

      public Map<String,String> getOperationMetrics()

    • setInCommitTimestamp

      public void setInCommitTimestamp(Optional<Long> inCommitTimestamp)

    • toRow

      public Row toRow()
      Encode as a Row object with the schema FULL_SCHEMA.
      Returns:
      Row object with the schema FULL_SCHEMA