Package io.delta.kernel.internal.util

Class SchemaUtils

Object
io.delta.kernel.internal.util.SchemaUtils
public class SchemaUtils extends Object
Utility methods for schema related operations such as validating the schema has no duplicate columns and the names contain only valid characters.

  • Method Details

    • validateSchema

      public static void validateSchema(StructType schema, boolean isColumnMappingEnabled, boolean isColumnDefaultEnabled, boolean isIcebergCompatV3Enabled)
      Validate the schema. This method checks if the schema has no duplicate columns, the names contain only valid characters, the data types are supported, and the column metadata is valid.
      Parameters:
      schema - the schema to validate
      isColumnMappingEnabled - whether column mapping is enabled. When column mapping is enabled, the column names in the schema can contain special characters that are allowed as column names in the Parquet file
      isColumnDefaultEnabled - whether column defaults is enabled
      Throws:
      IllegalArgumentException - if the schema is invalid

    • validateUpdatedSchemaAndGetUpdatedSchema

      public static Optional<StructType> validateUpdatedSchemaAndGetUpdatedSchema(Metadata currentMetadata, Metadata newMetadata, Protocol newProtocol, Set<String> clusteringColumnPhysicalNames, boolean allowNewRequiredFields)
      Performs the following validations on an updated table schema using the current schema as a base for validation. ColumnMapping must be enabled to call this.

      Returns an updated schema if metadata (i.e. TypeChanges needs to be copied over from currentSchema and new type changes need to be recorded. Kernel is expected to handle this work instead of clients).

      The following checks are performed:

      • No duplicate columns are allowed
      • Column names contain only valid characters
      • Data types are supported
      • Physical column name consistency is preserved in the new schema
      • If IcebergWriterCompatV1 is enabled, that map struct keys have not changed
      • ToDo: Nested IDs for array/map types are preserved in the new schema for IcebergCompatV2

    • validateSchemaEvolutionById

      public static Optional<StructType> validateSchemaEvolutionById(StructType currentSchema, StructType newSchema, Set<String> clusteringColumnPhysicalNames, int oldMaxFieldId, boolean allowNewRequiredFields, boolean icebergWriterCompatV1Enabled, boolean typeWideningEnabled)
      Validates a given schema evolution by using field ID as the source of truth for identifying fields
      Parameters:
      currentSchema - the schema that is present the table schema _before_ the schema evolution
      newSchema - the new schema that is present the table schema _after_ the schema evolution
      clusteringColumnPhysicalNames - The clustering columns present in the table before the schema update
      oldMaxFieldId - the maximum field id in the table before the schema update
      allowNewRequiredFields - If `false`, adding new required columns throws an error. If `true`, new required columns are allowed
      icebergWriterCompatV1Enabled - `true` if icebergCompatV1 is enabled on the table
      Returns:
      an updated schema if metadata (e.g. TypeChanges needs to be copied over from the old schema
      Throws:
      IllegalArgumentException - if the schema evolution is invalid

    • validatePartitionColumns

      public static void validatePartitionColumns(StructType schema, List<String> partitionCols)
      Verify the partition columns exists in the table schema and a supported data type for a partition column.
      Parameters:
      schema -
      partitionCols -

    • casePreservingPartitionColNames

      public static List<String> casePreservingPartitionColNames(StructType tableSchema, List<String> partitionColumns)
      Delta expects partition column names to be same case preserving as the name in the schema. E.g: Schema: (a INT, B STRING) and partition columns: (b). In this case we store the schema as (a INT, B STRING) and partition columns as (B).

      This method expects the inputs are already validated (i.e. schema contains all the partition columns).

    • casePreservingPartitionColNames

      public static Map<String,Literal> casePreservingPartitionColNames(List<String> partitionColNames, Map<String,Literal> partitionValues)
      Convert the partition column names in partitionValues map into the same case as the column in the table metadata. Delta expects the partition column names to preserve the case same as the table schema.
      Parameters:
      partitionColNames - List of partition columns in the table metadata. The names preserve the case as given by the connector when the table is created.
      partitionValues - Map of partition column name to partition value. Convert the partition column name to be same case preserving name as its equivalent column in the partitionColName. Column name comparison is case-insensitive.
      Returns:
      Rewritten partitionValues map with names case preserved.

    • casePreservingEligibleClusterColumns

      public static List<Column> casePreservingEligibleClusterColumns(StructType schema, List<Column> clusteringCols)
      Verify the clustering columns exists in the table schema.
      Parameters:
      schema - The schema of the table
      clusteringCols - List of clustering columns

    • findColIndex

      public static int findColIndex(StructType schema, String colName)
      Search (case-insensitive) for the given colName in the schema and return its position in the schema.
      Parameters:
      schema - StructType
      colName - Name of the column whose index is needed.
      Returns:
      Valid index or -1 if not found.

    • collectLeafColumns

      public static List<Column> collectLeafColumns(StructType schema, Set<String> excludedColumns, int maxColumns)
      Collects all leaf columns from the given schema (including flattened columns only for StructTypes), up to maxColumns. NOTE: If maxColumns = -1, we collect ALL leaf columns in the schema.

    • concatWithDot

      public static String concatWithDot(List<String> columnPath)
      Returns:
      column name by concatenating the column path elements (think of nested) with dots

    • asInternalColumn

      public static StructField asInternalColumn(StructField field)
      Helper method to create a copy of a column that is marked as an internal column.