Packages

o

org.apache.spark.sql.delta

DeltaOperations

object DeltaOperations

Exhaustive list of operations that can be performed on a Delta table. These operations are tracked as the first line in delta logs, and power DESCRIBE HISTORY for Delta tables.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DeltaOperations
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Type Members

  1. case class AddColumns(colsToAdd: Seq[QualifiedColTypeWithPositionForLog]) extends Operation with Product with Serializable

    Recorded when columns are added.

  2. case class AddConstraint(constraintName: String, expr: String) extends Operation with Product with Serializable
  3. case class ChangeColumn(columnPath: Seq[String], columnName: String, newColumn: StructField, colPosition: Option[String]) extends Operation with Product with Serializable

    Recorded when columns are changed.

  4. case class Clone(source: String, sourceVersion: Long) extends Operation with Product with Serializable
  5. case class ComputeStats(predicate: Seq[Expression]) extends OperationWithPredicates with Product with Serializable

    Recorded when recomputing stats on the table.

  6. case class Convert(numFiles: Long, partitionBy: Seq[String], collectStats: Boolean, catalogTable: Option[String], sourceFormat: Option[String]) extends Operation with Product with Serializable

    Recorded when converting a table into a Delta table.

  7. case class CreateTable(metadata: Metadata, isManaged: Boolean, asSelect: Boolean = false) extends Operation with Product with Serializable

    Recorded when the table is created.

  8. case class Delete(predicate: Seq[Expression]) extends OperationWithPredicates with Product with Serializable

    Recorded while deleting certain partitions.

  9. case class DropColumns(colsToDrop: Seq[Seq[String]]) extends Operation with Product with Serializable
  10. case class DropConstraint(constraintName: String, expr: Option[String]) extends Operation with Product with Serializable
  11. case class DropTableFeature(featureName: String, truncateHistory: Boolean) extends Operation with Product with Serializable

    Recorded when dropping a table feature.

  12. case class Merge(predicate: Option[Expression], updatePredicate: Option[String], deletePredicate: Option[String], insertPredicate: Option[String], matchedPredicates: Seq[MergePredicate], notMatchedPredicates: Seq[MergePredicate], notMatchedBySourcePredicates: Seq[MergePredicate]) extends OperationWithPredicates with Product with Serializable
  13. case class MergePredicate(predicate: Option[String], actionType: String) extends Product with Serializable

    Represents the predicates and action type (insert, update, delete) for a Merge clause

  14. sealed abstract class Operation extends AnyRef

    An operation that can be performed on a Delta table.

  15. abstract class OperationWithPredicates extends Operation
  16. case class Optimize(predicate: Seq[Expression], zOrderBy: Seq[String] = Seq.empty) extends OptimizeOrReorg with Product with Serializable

    Recorded when optimizing the table.

  17. sealed abstract class OptimizeOrReorg extends OperationWithPredicates
  18. case class QualifiedColTypeWithPositionForLog(columnPath: Seq[String], column: StructField, colPosition: Option[String]) extends Product with Serializable

    Qualified column type with position.

    Qualified column type with position. We define a copy of the type here to avoid depending on the parser output classes in our logging.

  19. case class RenameColumn(oldColumnPath: Seq[String], newColumnPath: Seq[String]) extends Operation with Product with Serializable
  20. case class Reorg(predicate: Seq[Expression], applyPurge: Boolean = true) extends OptimizeOrReorg with Product with Serializable

    Recorded when running REORG on the table.

  21. case class ReplaceColumns(columns: Seq[StructField]) extends Operation with Product with Serializable

    Recorded when columns are replaced.

  22. case class ReplaceTable(metadata: Metadata, isManaged: Boolean, orCreate: Boolean, asSelect: Boolean = false, userMetadata: Option[String] = None) extends Operation with Product with Serializable

    Recorded when the table is replaced.

  23. case class Restore(version: Option[Long], timestamp: Option[String]) extends Operation with Product with Serializable
  24. case class SetTableProperties(properties: Map[String, String]) extends Operation with Product with Serializable
  25. case class StreamingUpdate(outputMode: OutputMode, queryId: String, epochId: Long, userMetadata: Option[String] = None) extends Operation with Product with Serializable

    Recorded during streaming inserts.

  26. case class TestOperation(operationName: String = "TEST") extends Operation with Product with Serializable

    Dummy operation only for testing with arbitrary operation names

  27. case class Truncate() extends Operation with Product with Serializable

    Recorded when truncating the table.

  28. case class UnsetTableProperties(propKeys: Seq[String], ifExists: Boolean) extends Operation with Product with Serializable

    Recorded when the table properties are unset.

  29. case class Update(predicate: Option[Expression]) extends OperationWithPredicates with Product with Serializable

    Recorded when an update operation is committed to the table.

  30. case class UpdateColumnMetadata(operationName: String, columns: Seq[(Seq[String], StructField)]) extends Operation with Product with Serializable
  31. case class UpdateSchema(oldSchema: StructType, newSchema: StructType) extends Operation with Product with Serializable
  32. case class UpgradeProtocol(newProtocol: Protocol) extends Operation with Product with Serializable
  33. case class VacuumEnd(status: String) extends Operation with Product with Serializable

    status

    - whether the vacuum operation was successful; either "COMPLETED" or "FAILED"

  34. case class VacuumStart(retentionCheckEnabled: Boolean, specifiedRetentionMillis: Option[Long], defaultRetentionMillis: Long) extends Operation with Product with Serializable

    retentionCheckEnabled

    - whether retention check was enabled for this run of vacuum.

    specifiedRetentionMillis

    - specified retention interval

    defaultRetentionMillis

    - default retention period for the table

  35. case class Write(mode: SaveMode, partitionBy: Option[Seq[String]] = None, predicate: Option[String] = None, userMetadata: Option[String] = None) extends Operation with Product with Serializable

    Recorded during batch inserts.

    Recorded during batch inserts. Predicates can be provided for overwrites.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. val OPTIMIZE_OPERATION_NAME: String

    operation name for OPTIMIZE command

  5. val OP_CLONE: String

    Recorded when cloning a Delta table into a new location.

  6. val OP_DROP_COLUMN: String

    Recorded when columns are dropped.

  7. val OP_MERGE: String

    Recorded when a merge operation is committed to the table.

    Recorded when a merge operation is committed to the table.

    updatePredicate, deletePredicate, and insertPredicate are DEPRECATED. Only use predicate, matchedPredicates, notMatchedPredicates and notMatchedBySourcePredicates to record the merge.

  8. val OP_RENAME_COLUMN: String

    Recorded when column is renamed

  9. val OP_RESTORE: String

    Recorded when restoring a Delta table to an older version.

  10. val OP_SET_TBLPROPERTIES: String

    Recorded when the table properties are set.

  11. val REORG_OPERATION_NAME: String

    operation name for REORG command

  12. val ZORDER_PARAMETER_KEY: String

    parameter key to indicate which columns to z-order by

  13. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  14. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  15. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  16. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  17. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  18. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  19. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  20. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  21. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  22. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  23. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  24. def predicatesToString(predicates: Seq[Expression]): Seq[String]

    Helper method to convert a sequence of command predicates in the form of an Expressions to a sequence of Strings so be stored in the commit info.

  25. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  26. def toString(): String
    Definition Classes
    AnyRef → Any
  27. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  28. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  29. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  30. object EmptyCommit extends Operation

    A commit without any actions.

    A commit without any actions. Could be used to force creation of new checkpoints.

  31. object ManualUpdate extends Operation
  32. object Merge extends Serializable
  33. object MergePredicate extends Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped