Package io.delta.kernel.internal.files

Class ParsedLogData

Object
io.delta.kernel.internal.files.ParsedLogData
Direct Known Subclasses:
ParsedCheckpointData, ParsedChecksumData, ParsedDeltaData, ParsedLogCompactionData
public abstract class ParsedLogData extends Object
Abstract representation of any valid log type in the Delta log.

Different child classes are used to represent the different log types.

Any given log type can be written as a file or represented inline and given to Kernel as a ColumnarBatch. That is: Kernel just needs to know how to parse and interpret a given log type (Kernel will of course treat Deltas differently than Checksums) as well as how to get that log type's bytes. This is why a given log type can be represented as either a file or inline.

For now, our APIs only allow creating ParsedCatalogCommitData inline, but we may change and expand this capability in the future.

The supported log types are:

  • Published Deltas: 00000000000000000001.json
  • Catalog Commits: _staged_commits/00000000000000000001.uuid-1234.json
  • Log compaction files: 00000000000000000001.00000000000000000009.compacted.json
  • Checksum files: 00000000000000000001.crc
  • Classic Checkpoint files: 00000000000000000001.checkpoint.parquet
  • V2 checkpoint files: 00000000000000000001.checkpoint.uuid-1234.json
  • Multi-part checkpoint files: 00000000000000000001.checkpoint.0000000001.0000000010.parquet

  • Method Details

    • forFileStatus

      public static ParsedLogData forFileStatus(FileStatus fileStatus)

    • isFile

      public boolean isFile()
      Returns true if this log data is stored as a file on disk. When false, the data is stored inline.

    • isInline

      public boolean isInline()
      Returns true if this log data is stored inline as a ColumnarBatch. When false, the data is stored as a file on disk.

    • getVersion

      public long getVersion()
      Return the version of this log data.

    • getFileStatus

      public FileStatus getFileStatus()
      Callers must check isFile() before calling this method.
      Throws:
      NoSuchElementException - if isFile() is false

    • getInlineData

      public ColumnarBatch getInlineData()
      Callers must check isInline() before calling this method.
      Throws:
      NoSuchElementException - if isInline() is false

    • getGroupByCategoryClass

      public abstract Class<? extends ParsedLogData> getGroupByCategoryClass()
      Returns the category class used for grouping collections of LISTed ParsedLogData instances.

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class Object

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Object

    • toString

      public String toString()
      Overrides:
      toString in class Object