Package io.delta.kernel.internal

Class TableConfig<T>

Object
io.delta.kernel.internal.TableConfig<T>
public class TableConfig<T> extends Object
Represents the table properties. Also provides methods to access the property values from the table metadata.

  • Field Details

    • MIN_PROTOCOL_READER_VERSION_KEY

      public static final String MIN_PROTOCOL_READER_VERSION_KEY
      See Also:

    • MIN_PROTOCOL_WRITER_VERSION_KEY

      public static final String MIN_PROTOCOL_WRITER_VERSION_KEY
      See Also:

    • APPEND_ONLY_ENABLED

      public static final TableConfig<Boolean> APPEND_ONLY_ENABLED
      Whether this Delta table is append-only. Files can't be deleted, or values can't be updated.

    • CHANGE_DATA_FEED_ENABLED

      public static final TableConfig<Boolean> CHANGE_DATA_FEED_ENABLED
      Enable change data feed output. When enabled, DELETE, UPDATE, and MERGE INTO operations will need to do additional work to output their change data in an efficiently readable format.

    • CHECKPOINT_POLICY

      public static final TableConfig<String> CHECKPOINT_POLICY

    • DELETION_VECTORS_CREATION_ENABLED

      public static final TableConfig<Boolean> DELETION_VECTORS_CREATION_ENABLED
      Whether commands modifying this Delta table are allowed to create new deletion vectors.

    • TYPE_WIDENING_ENABLED

      public static final TableConfig<Boolean> TYPE_WIDENING_ENABLED
      Whether widening the type of an existing column or field is allowed, either manually using ALTER TABLE CHANGE COLUMN or automatically if automatic schema evolution is enabled.

    • ROW_TRACKING_ENABLED

      public static final TableConfig<Boolean> ROW_TRACKING_ENABLED
      Indicates whether Row Tracking is enabled on the table. When this flag is turned on, all rows are guaranteed to have Row IDs and Row Commit Versions assigned to them, and writers are expected to preserve them by materializing them to hidden columns in the data files.

    • TOMBSTONE_RETENTION

      public static final TableConfig<Long> TOMBSTONE_RETENTION
      The shortest duration we have to keep logically deleted data files around before deleting them physically.

      Note: this value should be large enough:

      • It should be larger than the longest possible duration of a job if you decide to run "VACUUM" when there are concurrent readers or writers accessing the table.
      • If you are running a streaming query reading from the table, you should make sure the query doesn't stop longer than this value. Otherwise, the query may not be able to restart as it still needs to read old files.

    • CHECKPOINT_INTERVAL

      public static final TableConfig<Integer> CHECKPOINT_INTERVAL
      How often to checkpoint the delta log? For every N (this config) commits to the log, we will suggest write out a checkpoint file that can speed up the Delta table state reconstruction.

    • LOG_RETENTION

      public static final TableConfig<Long> LOG_RETENTION
      The shortest duration we have to keep delta/checkpoint files around before deleting them. We can only delete delta files that are before a checkpoint.

    • EXPIRED_LOG_CLEANUP_ENABLED

      public static final TableConfig<Boolean> EXPIRED_LOG_CLEANUP_ENABLED
      Whether to clean up expired checkpoints and delta logs.

    • IN_COMMIT_TIMESTAMPS_ENABLED

      public static final TableConfig<Boolean> IN_COMMIT_TIMESTAMPS_ENABLED
      This table property is used to track the enablement of the inCommitTimestamps.

      When enabled, commit metadata includes a monotonically increasing timestamp that allows for reliable TIMESTAMP AS OF time travel even if filesystem operations change a commit file's modification timestamp.

    • IN_COMMIT_TIMESTAMP_ENABLEMENT_VERSION

      public static final TableConfig<Optional<Long>> IN_COMMIT_TIMESTAMP_ENABLEMENT_VERSION
      This table property is used to track the version of the table at which inCommitTimestamps were enabled.

    • IN_COMMIT_TIMESTAMP_ENABLEMENT_TIMESTAMP

      public static final TableConfig<Optional<Long>> IN_COMMIT_TIMESTAMP_ENABLEMENT_TIMESTAMP
      This table property is used to track the timestamp at which inCommitTimestamps were enabled. More specifically, it is the inCommitTimestamps of the commit with the version specified in IN_COMMIT_TIMESTAMP_ENABLEMENT_VERSION.

    • COLUMN_MAPPING_MODE

      public static final TableConfig<ColumnMapping.ColumnMappingMode> COLUMN_MAPPING_MODE
      This table property is used to control the column mapping mode.

    • COLUMN_MAPPING_MAX_COLUMN_ID

      public static final TableConfig<Long> COLUMN_MAPPING_MAX_COLUMN_ID
      This table property is used to control the maximum column mapping ID.

    • ICEBERG_COMPAT_V2_ENABLED

      public static final TableConfig<Boolean> ICEBERG_COMPAT_V2_ENABLED
      Table property that enables modifying the table in accordance with the Delta-Iceberg Compatibility V2 protocol.
      See Also:

    • ICEBERG_COMPAT_V3_ENABLED

      public static final TableConfig<Boolean> ICEBERG_COMPAT_V3_ENABLED
      Table property that enables modifying the table in accordance with the Delta-Iceberg Compatibility V3 protocol. TODO: add the delta protocol link once updated [https://github.com/delta-io/delta/issues/4574]

    • DATA_SKIPPING_NUM_INDEXED_COLS

      public static final TableConfig<Integer> DATA_SKIPPING_NUM_INDEXED_COLS
      The number of columns to collect stats on for data skipping. A value of -1 means collecting stats for all columns.

      For Struct types, all leaf fields count individually toward this limit in depth-first order. For example, if a table has columns a, b.c, b.d, and e, then the first three indexed columns would be a, b.c, and b.d. Map and array types are not supported for statistics collection.

    • DATA_SKIPPING_STATS_COLUMNS

      public static final TableConfig<Optional<String>> DATA_SKIPPING_STATS_COLUMNS
      IMPORTANT: This table property is recognized but is not yet validated, enforced, or implemented by Kernel.

      The names of specific columns to collect stats on for data skipping. If present, it takes precedence over DATA_SKIPPING_NUM_INDEXED_COLS, and the system will only collect stats for columns that exactly match those specified. If a nested column is specified, the system will collect stats for all leaf fields of that column. If a non-existent column is specified, it will be ignored. Updating this config does not trigger stats re-collection, but redefines the stats schema of the table, i.e., it will change the behavior of future stats collection (e.g., in append and OPTIMIZE) as well as data skipping (e.g., the column stats not mentioned by this config will be ignored even if they exist).

      The value is a comma-separated list of case-insensitive column identifiers. Each column identifier can consist of letters, digits, and underscores. If a column identifier includes special characters, the column name should be enclosed in backticks (`) to escape the special characters.

      A column identifier can refer to one of the following: the name of a non-struct column, the leaf field's name of a struct column, or the name of a struct column. When a struct column's name is specified, statistics for all its leaf fields will be collected.

    • ICEBERG_WRITER_COMPAT_V1_ENABLED

      public static final TableConfig<Boolean> ICEBERG_WRITER_COMPAT_V1_ENABLED
      Table property that enables modifying the table in accordance with the Delta-Iceberg Writer Compatibility V1 (icebergCompatWriterV1) protocol.

    • ICEBERG_WRITER_COMPAT_V3_ENABLED

      public static final TableConfig<Boolean> ICEBERG_WRITER_COMPAT_V3_ENABLED
      Table property that enables modifying the table in accordance with the Delta-Iceberg Writer Compatibility V3 (icebergCompatWriterV3) protocol. V2 is skipped to align with the iceberg v3 spec.

    • UNIVERSAL_FORMAT_ENABLED_FORMATS

      public static final TableConfig<Set<String>> UNIVERSAL_FORMAT_ENABLED_FORMATS
      Table config that allows for translation of Delta metadata to other table formats metadata.

    • VARIANT_SHREDDING_ENABLED

      public static final TableConfig<Boolean> VARIANT_SHREDDING_ENABLED
      Table property that enables modifying the table in accordance with the Delta-Variant Shredding protocol.
      See Also:

    • MATERIALIZED_ROW_ID_COLUMN_NAME

      public static final TableConfig<String> MATERIALIZED_ROW_ID_COLUMN_NAME

    • MATERIALIZED_ROW_COMMIT_VERSION_COLUMN_NAME

      public static final TableConfig<String> MATERIALIZED_ROW_COMMIT_VERSION_COLUMN_NAME

  • Method Details

    • validateAndNormalizeDeltaProperties

      public static Map<String,String> validateAndNormalizeDeltaProperties(Map<String,String> newProperties)
      Validates that the given new properties that the txn is trying to update in table. Properties that have `delta.` prefix in the key name should be in valid list and are editable. The caller is expected to store the returned properties in the table metadata after further validation from a protocol point of view. The returned properties will have the key's case normalized as defined in its TableConfig.
      Parameters:
      newProperties - the properties to validate
      Throws:
      InvalidConfigurationValueException - if any of the properties are invalid
      UnknownConfigurationException - if any of the properties are unknown

    • fromMetadata

      public T fromMetadata(Metadata metadata)
      Returns the value of the table property from the given metadata.
      Parameters:
      metadata - the table metadata
      Returns:
      the value of the table property

    • fromMetadata

      public T fromMetadata(Map<String,String> configuration)
      Returns the value of the table property from the given configuration.
      Parameters:
      configuration - the table configuration
      Returns:
      the value of the table property

    • getKey

      public String getKey()
      Returns the key of the table property.
      Returns:
      the key of the table property