Package io.delta.kernel.internal.rowtracking

Class MaterializedRowTrackingColumn

Object
io.delta.kernel.internal.rowtracking.MaterializedRowTrackingColumn
public final class MaterializedRowTrackingColumn extends Object
A collection of helper methods for working with materialized row tracking columns.

  • Field Details

    • MATERIALIZED_ROW_ID

      public static final MaterializedRowTrackingColumn MATERIALIZED_ROW_ID
      Static instance for the materialized row ID column.

    • MATERIALIZED_ROW_COMMIT_VERSION

      public static final MaterializedRowTrackingColumn MATERIALIZED_ROW_COMMIT_VERSION
      Static instance for the materialized row commit version column.

  • Method Details

    • getMaterializedColumnNameProperty

      public String getMaterializedColumnNameProperty()
      Returns the configuration property name associated with this materialized column.

    • getMaterializedColumnNamePrefix

      public String getMaterializedColumnNamePrefix()
      Returns the prefix to use for generating the materialized column name.

    • throwIfColumnNamesConflictWithSchema

      public static void throwIfColumnNamesConflictWithSchema(Metadata metadata)
      Validates that the materialized column names for ROW_ID and ROW_COMMIT_VERSION do not conflict with any existing logical or physical column names in the table's schema.
      Parameters:
      metadata - The current Metadata of the table.

    • validateRowTrackingConfigsNotMissing

      public static void validateRowTrackingConfigsNotMissing(Metadata metadata, String tablePath)
      Validates that materialized column names for ROW_ID and ROW_COMMIT_VERSION are not missing when row tracking is enabled. This should be called for existing tables to ensure that row tracking configs are present when they should be.
      Parameters:
      metadata - The current Metadata of the table.

    • assignMaterializedColumnNamesIfNeeded

      public static Optional<Metadata> assignMaterializedColumnNamesIfNeeded(Metadata metadata)
      Assigns materialized column names for ROW_ID and ROW_COMMIT_VERSION if row tracking is enabled and the column names have not been assigned yet.
      Parameters:
      metadata - The current Metadata of the table.
      Returns:
      An Optional containing updated metadata if any assignments occurred; Optional.empty() otherwise.

    • convertToPhysicalColumn

      public static List<StructField> convertToPhysicalColumn(StructField logicalField, StructType logicalSchema, Metadata metadata)
      Converts a logical row tracking field to its physical counterpart(s).

      Since computing the row ID requires the row index, requesting a row tracking column can require adding two columns to the physical schema.

      Note that we must not mark the physical columns as metadata columns because as far as the parquet reader is concerned, these columns are not metadata columns.

      Parameters:
      logicalField - The logical field to convert.
      logicalSchema - The logical schema containing the field.
      metadata - The current metadata of the table.
      Returns:
      A list of StructFields representing the physical columns corresponding to the logical field.

    • transformPhysicalData

      public static ColumnarBatch transformPhysicalData(ColumnarBatch dataBatch, Row scanFile, StructType logicalSchema, Map<String,String> configuration, Engine engine)
      Computes row IDs and row commit versions based on their materialized values if present in the data returned by the Parquet reader, using the base row ID and default row commit version from the AddFile otherwise.
      Parameters:
      dataBatch - a batch of physical data read from the table.
      scanFile - the Row representing the scan file metadata.
      logicalSchema - the logical schema of the query.
      configuration - the configuration map containing table metadata.
      engine - the Engine to use for expression evaluation.
      Returns:
      a new ColumnarBatch with logical row tracking columns

    • getPhysicalColumnName

      public String getPhysicalColumnName(Map<String,String> configuration)
      Gets the physical column name from the table configuration.
      Parameters:
      configuration - the table configuration map
      Returns:
      the physical column name
      Throws:
      IllegalArgumentException - if the materialized column name is missing from the configuration