Package io.delta.kernel.internal.actions

Class GenerateIcebergCompatActionUtils

Object
io.delta.kernel.internal.actions.GenerateIcebergCompatActionUtils
public final class GenerateIcebergCompatActionUtils extends Object
Utilities to convert Iceberg add/removes to Delta Kernel add/removes

  • Constructor Details

    • GenerateIcebergCompatActionUtils

      public GenerateIcebergCompatActionUtils()

  • Method Details

    • generateIcebergCompatWriterV1AddAction

      public static Row generateIcebergCompatWriterV1AddAction(Row transactionState, DataFileStatus fileStatus, Map<String,Literal> partitionValues, boolean dataChange, Map<String,String> tags, Optional<StructType> physicalSchemaOpt)
      Create an add action Row that can be passed to Transaction.commit(Engine, CloseableIterable) from an Iceberg add.
      Parameters:
      transactionState - the transaction state from the built transaction
      fileStatus - the file status to create the add with (contains path, time, size, and stats)
      partitionValues - the partition values for the add
      dataChange - whether or not the add constitutes a dataChange (i.e. append vs. compaction)
      tags - key-value metadata to be attached to the add action
      physicalSchemaOpt - An optional pre-parsed physical schema. Improves performance for batch operations by avoiding repeated JSON parsing. Recommended when generating many actions with the same schema.
      Returns:
      add action row that can be included in the transaction
      Throws:
      UnsupportedOperationException - if icebergWriterCompatV1 is not enabled
      UnsupportedOperationException - if maxRetries != 0 in the transaction
      KernelException - if stats are not present (required for icebergCompatV2)
      UnsupportedOperationException - if the table is partitioned (currently unsupported)

    • generateIcebergCompatWriterV1AddAction

      public static Row generateIcebergCompatWriterV1AddAction(Row transactionState, DataFileStatus fileStatus, Map<String,Literal> partitionValues, boolean dataChange, Optional<StructType> physicalSchemaOpt)

    • generateIcebergCompatWriterV3AddAction

      public static Row generateIcebergCompatWriterV3AddAction(Row transactionState, DataFileStatus fileStatus, Map<String,Literal> partitionValues, boolean dataChange, Map<String,String> tags, Optional<Long> baseRowId, Optional<Long> defaultRowCommitVersion, Optional<DeletionVectorDescriptor> deletionVectorDescriptor, Optional<StructType> physicalSchemaOpt)
      Create an add action Row that can be passed to Transaction.commit(Engine, CloseableIterable) from an Iceberg add.
      Parameters:
      transactionState - the transaction state from the built transaction
      fileStatus - the file status to create the add with (contains path, time, size, and stats)
      partitionValues - the partition values for the add
      dataChange - whether or not the add constitutes a dataChange (i.e. append vs. compaction)
      tags - key-value metadata to be attached to the add action
      deletionVectorDescriptor - optional deletion vector descriptor for the add action
      physicalSchemaOpt - An optional pre-parsed physical schema. Improves performance for batch operations by avoiding repeated JSON parsing. Recommended when generating many actions with the same schema.
      Returns:
      add action row that can be included in the transaction
      Throws:
      UnsupportedOperationException - if icebergWriterCompatV3 is not enabled
      UnsupportedOperationException - if maxRetries != 0 in the transaction
      KernelException - if stats are not present (required for icebergCompatV3)
      UnsupportedOperationException - if the table is partitioned (currently unsupported)

    • generateIcebergCompatWriterV1RemoveAction

      public static Row generateIcebergCompatWriterV1RemoveAction(Row transactionState, DataFileStatus fileStatus, Map<String,Literal> partitionValues, boolean dataChange, Optional<StructType> physicalSchemaOpt)
      Create a remove action Row that can be passed to Transaction.commit(Engine, CloseableIterable) from an Iceberg remove.
      Parameters:
      transactionState - the transaction state from the built transaction
      fileStatus - the file status to create the remove with (contains path, time, size, and stats)
      partitionValues - the partition values for the remove
      dataChange - whether or not the remove constitutes a dataChange (i.e. delete vs. compaction)
      physicalSchemaOpt - An optional pre-parsed physical schema. Improves performance for batch operations by avoiding repeated JSON parsing. Recommended when generating many actions with the same schema.
      Returns:
      remove action row that can be committed to the transaction
      Throws:
      UnsupportedOperationException - if icebergWriterCompatV1 is not enabled
      UnsupportedOperationException - if maxRetries != 0 in the transaction
      KernelException - if the table is an append-only table and dataChange=true
      UnsupportedOperationException - if the table is partitioned (currently unsupported)

    • generateIcebergCompatWriterV3RemoveAction

      public static Row generateIcebergCompatWriterV3RemoveAction(Row transactionState, DataFileStatus fileStatus, Map<String,Literal> partitionValues, boolean dataChange, Optional<Long> baseRowId, Optional<Long> defaultRowCommitVersion, Optional<DeletionVectorDescriptor> deletionVectorDescriptor, Optional<StructType> physicalSchemaOpt)
      Create a remove action Row that can be passed to Transaction.commit(Engine, CloseableIterable) from an Iceberg remove.
      Parameters:
      transactionState - the transaction state from the built transaction
      fileStatus - the file status to create the remove with (contains path, time, size, and stats)
      partitionValues - the partition values for the remove
      dataChange - whether or not the remove constitutes a dataChange (i.e. delete vs. compaction)
      deletionVectorDescriptor - optional deletion vector descriptor for the add action
      physicalSchemaOpt - An optional pre-parsed physical schema. Improves performance for batch operations by avoiding repeated JSON parsing. Recommended when generating many actions with the same schema.
      Returns:
      remove action row that can be committed to the transaction
      Throws:
      UnsupportedOperationException - if icebergWriterCompatV3 is not enabled
      UnsupportedOperationException - if maxRetries != 0 in the transaction
      KernelException - if the table is an append-only table and dataChange=true
      UnsupportedOperationException - if the table is partitioned (currently unsupported)

    • convertRemoveDataFileStatus

      @VisibleForTesting public static Row convertRemoveDataFileStatus(StructType physicalSchema, URI tableRoot, DataFileStatus dataFileStatus, Map<String,Literal> partitionValues, boolean dataChange, Optional<Long> baseRowId, Optional<Long> defaultRowCommitVersion, Optional<DeletionVectorDescriptor> deletionVectorDescriptor)

    • createRemoveFileRowWithExtendedFileMetadata

      @VisibleForTesting public static Row createRemoveFileRowWithExtendedFileMetadata(String path, long deletionTimestamp, boolean dataChange, MapValue partitionValues, long size, Optional<DataFileStatistics> stats, StructType physicalSchema, Optional<Long> baseRowId, Optional<Long> defaultRowCommitVersion, Optional<DeletionVectorDescriptor> deletionVector)