Package io.delta.kernel.internal.icebergcompat

Class IcebergWriterCompatV1MetadataValidatorAndUpdater

Object
io.delta.kernel.internal.icebergcompat.IcebergCompatMetadataValidatorAndUpdater
io.delta.kernel.internal.icebergcompat.IcebergWriterCompatV1MetadataValidatorAndUpdater
public class IcebergWriterCompatV1MetadataValidatorAndUpdater extends IcebergCompatMetadataValidatorAndUpdater
Performs the validations and updates necessary to support the table feature IcebergWriterCompatV1 when it is enabled by the table property "delta.enableIcebergWriterCompatV1".

Requires that the following table properties are set to the specified values. If they are set to an invalid value, throws an exception. If they are not set, enable them if possible.

  • Requires ID column mapping mode (cannot be enabled on existing table).
  • Requires icebergCompatV2 to be enabled.

Checks that required table features are enabled: icebergCompatWriterV1, icebergCompatV2, columnMapping

Checks the following:

  • Checks that all table features supported in the table's protocol are in the allow-list of table features. This simultaneously ensures that any unsupported features are not present (e.g. CDF, variant type, etc).
  • Checks that there are no fields with data type byte or short.
  • Checks that the table feature `invariants` is not active in the table (i.e. there are no invariants in the table schema). This is a special case where the incompatible feature `invariants` is in the allow-list of features since it is included by default in the table protocol for new tables. Since it is incompatible we must verify that it is inactive in the table.
TODO additional enforcements coming in (ie physicalName=fieldId)

  • Constructor Details

    • IcebergWriterCompatV1MetadataValidatorAndUpdater

      public IcebergWriterCompatV1MetadataValidatorAndUpdater()

  • Method Details

    • validateIcebergWriterCompatV1Change

      public static void validateIcebergWriterCompatV1Change(Map<String,String> oldConfig, Map<String,String> newConfig)
      Validates that any change to property TableConfig.ICEBERG_WRITER_COMPAT_V1_ENABLED is valid (for existing table). Currently, the changes we support are
      • No change in enablement (true to true or false to false)
      The changes that we do not support and for which we throw an KernelException are
      • Disabling on an existing table (true to false)
      • Enabling on an existing table (false to true)

    • validateAndUpdateIcebergWriterCompatV1Metadata

      public static Optional<Metadata> validateAndUpdateIcebergWriterCompatV1Metadata(boolean isCreatingNewTable, Metadata newMetadata, Protocol newProtocol, Optional<Protocol> prevProtocol)
      Validate and update the given Iceberg Writer Compat V1 metadata.
      Parameters:
      newMetadata - Metadata after the current updates
      newProtocol - Protocol after the current updates
      Returns:
      The updated metadata if the metadata is valid and updated, otherwise empty.
      Throws:
      UnsupportedOperationException - if the metadata is not compatible with Iceberg Writer V1 requirements