Package io.delta.kernel.internal.util

Class ColumnMapping

Object
io.delta.kernel.internal.util.ColumnMapping
public class ColumnMapping extends Object
Utilities related to the column mapping feature.

  • Field Details

  • Method Details

    • getColumnMappingMode

      public static ColumnMapping.ColumnMappingMode getColumnMappingMode(Map<String,String> configuration)
      Returns the column mapping mode from the given configuration.
      Parameters:
      configuration - Configuration
      Returns:
      Column mapping mode. One of ("none", "name", "id")

    • convertToPhysicalSchema

      public static StructType convertToPhysicalSchema(StructType prunedSchema, StructType fullSchema, ColumnMapping.ColumnMappingMode columnMappingMode)
      Helper method that converts the logical schema (requested by the connector) to physical schema of the data stored in data files based on the table's column mapping mode. Field-id column metadata is preserved when cmMode = ID, all column metadata is otherwise removed.

      We require fullSchema in addition to the pruned schema we want to convert since we need the complete field metadata as it is stored in the schema in the _delta_log. We cannot be sure (and do not enforce) that this metadata is preserved by the connector.

      Parameters:
      prunedSchema - the logical read schema requested by the connector
      fullSchema - the full delta schema (with complete metadata) as read from the _delta_log
      columnMappingMode - Column mapping mode

    • convertToPhysicalColumn

      public static StructField convertToPhysicalColumn(StructField logicalField, StructType fullSchema, ColumnMapping.ColumnMappingMode columnMappingMode)
      Converts a logical column to a physical column based on the table's column mapping mode. The field-id metadata is preserved when cmMode = ID, all column metadata is otherwise removed.

      We require fullSchema in addition to the logical field we want to convert since we need the complete field metadata as it is stored in the schema in the _delta_log. We cannot be sure (and do not enforce) that this metadata is preserved by the connector.

      Parameters:
      logicalField - the logical read column requested by the connector
      fullSchema - the full delta schema (with complete metadata) as read from the _delta_log
      columnMappingMode - Column mapping mode

    • getPhysicalName

      public static String getPhysicalName(StructField field)
      Returns the physical name for a given StructField

    • getColumnId

      public static int getColumnId(StructField field)
      Returns the column id for a given StructField

    • verifyColumnMappingChange

      public static void verifyColumnMappingChange(Map<String,String> oldConfig, Map<String,String> newConfig)

    • isColumnMappingModeEnabled

      public static boolean isColumnMappingModeEnabled(ColumnMapping.ColumnMappingMode columnMappingMode)

    • updateColumnMappingMetadataIfNeeded

      public static Optional<Metadata> updateColumnMappingMetadataIfNeeded(Metadata metadata, boolean isNewTable)
      Updates the column mapping metadata if needed based on the column mapping mode and whether the icebergCompatV2 is enabled. If column mapping/iceberg compat info is already present in the metadata, this method does nothing and returns an empty Optional. Callers can avoid updating the metadata if the metadata has not changed.
      Parameters:
      metadata - Current metadata.
      isNewTable - Whether this is part of a commit that sets the mapping mode on a new table.
      Returns:
      Optional of the updated metadata if it has changed, Optional.empty() otherwise.

    • getPhysicalColumnNameAndDataType

      public static Tuple2<Column,DataType> getPhysicalColumnNameAndDataType(StructType schema, Column logicalColumn)
      Returns the physical column and data type for a given logical column based on the schema.

    • getLogicalColumnNameAndDataType

      public static Tuple2<Column,DataType> getLogicalColumnNameAndDataType(StructType schema, Column physicalColumn)
      Returns the logical column and data type for a given physical column based on the schema.

    • blockIfColumnMappingEnabled

      public static void blockIfColumnMappingEnabled(Row transactionState)
      Utility method to block writing into a table with column mapping enabled. Currently Kernel only supports the metadata updates on tables with column mapping enabled. Data writes into such tables using the data transformation APIs provided by the Kernel are not supported yet.