Value Members

1.  final def !=(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

2.  final def ##: Int

   Definition Classes

   AnyRef → Any

3.  final def ==(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

4.  val COLUMN_MAPPING_METADATA_ID_KEY: String
5.  val COLUMN_MAPPING_METADATA_PREFIX: String
6.  val COLUMN_MAPPING_PHYSICAL_NAME_KEY: String
7.  val DELTA_INTERNAL_COLUMNS: Set[String]

   This list of internal columns (and only this list) is allowed to have missing column mapping metadata such as field id and physical name because they might not be present in user's table schema.

   This list of internal columns (and only this list) is allowed to have missing column mapping metadata such as field id and physical name because they might not be present in user's table schema.

   These fields, if materialized to parquet, will always be matched by their display name in the downstream parquet reader even under column mapping modes.

   For future developers who want to utilize additional internal columns without generating column mapping metadata, please add them here.

   This list is case-insensitive.

   Attributes

   protected

8.  val PARQUET_FIELD_ID_METADATA_KEY: String
9.  final def asInstanceOf[T0]: T0

   Definition Classes

   Any

10.  def assignColumnIdAndPhysicalName(newMetadata: Metadata, oldMetadata: Metadata, isChangingModeOnExistingTable: Boolean, isOverwritingSchema: Boolean): Metadata

    For each column/field in a Metadata's schema, assign id using the current maximum id as the basis and increment from there, and assign physical name using UUID

    For each column/field in a Metadata's schema, assign id using the current maximum id as the basis and increment from there, and assign physical name using UUID

    newMetadata

    The new metadata to assign Ids and physical names

    oldMetadata

    The old metadata

    isChangingModeOnExistingTable

    whether this is part of a commit that changes the mapping mode on a existing table

    returns

    new metadata with Ids and physical names assigned

11.  def assignPhysicalName(field: StructField, physicalName: String): StructField
12.  def assignPhysicalNames(schema: StructType): StructType
13.  def checkColumnIdAndPhysicalNameAssignments(metadata: Metadata): Unit

    Verify the metadata for valid column mapping metadata assignment.

    Verify the metadata for valid column mapping metadata assignment. This is triggered for every commit as a last defense.

    1. Ensure column mapping metadata is set for the appropriate mode 2. Ensure no duplicate column id/physical names set 3. Ensure max column id is in a good state (set, and greater than all field ids available)

14.  def clone(): AnyRef

    Attributes

    protected[lang]

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.CloneNotSupportedException]) @native()

15.  def createPhysicalAttributes(output: Seq[Attribute], referenceSchema: StructType, columnMappingMode: DeltaColumnMappingMode): Seq[Attribute]

    Create a list of physical attributes for the given attributes using the table schema as a reference.

    Create a list of physical attributes for the given attributes using the table schema as a reference.

    output

    the list of attributes (potentially without any metadata)

    referenceSchema

    the table schema with all the metadata

    columnMappingMode

    column mapping mode of the delta table, which determines which metadata to fill in

16.  def createPhysicalSchema(schema: StructType, referenceSchema: StructType, columnMappingMode: DeltaColumnMappingMode, checkSupportedMode: Boolean = true): StructType

    Create a physical schema for the given schema using the Delta table schema as a reference.

    Create a physical schema for the given schema using the Delta table schema as a reference.

    schema

    the given logical schema (potentially without any metadata)

    referenceSchema

    the schema from the delta log, which has all the metadata

    columnMappingMode

    column mapping mode of the delta table, which determines which metadata to fill in

    checkSupportedMode

    whether we should check of the column mapping mode is supported

17.  def dropColumnMappingMetadata(schema: StructType): StructType
18.  final def eq(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

19.  def equals(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef → Any

20.  def filterColumnMappingProperties(properties: Map[String, String]): Map[String, String]
21.  def finalize(): Unit

    Attributes

    protected[lang]

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.Throwable])

22.  def findMaxColumnId(schema: StructType): Long
23.  def generatePhysicalName: String
24.  final def getClass(): Class[_ <: AnyRef]

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

25.  def getColumnId(field: StructField): Int
26.  def getColumnMappingMetadata(field: StructField, mode: DeltaColumnMappingMode): Metadata

    Gets the required column metadata for each column based on the column mapping mode.

27.  def getCommonTags(deltaLog: DeltaLog, tahoeId: String): Map[TagDefinition, String]

    Definition Classes

    DeltaLogging

28.  def getPhysicalName(field: StructField): String
29.  def getPhysicalNameFieldMap(schema: StructType): Map[Seq[String], StructField]

    Returns a map of physicalNamePath -> field for the given schema, where physicalNamePath is the [$parentPhysicalName, ..., $fieldPhysicalName] list of physical names for every field (including nested) in the schema.

    Returns a map of physicalNamePath -> field for the given schema, where physicalNamePath is the [$parentPhysicalName, ..., $fieldPhysicalName] list of physical names for every field (including nested) in the schema.

    Must be called after checkColumnIdAndPhysicalNameAssignments, so that we know the schema is valid.

30.  def hasColumnId(field: StructField): Boolean
31.  def hasNoColumnMappingSchemaChanges(newMetadata: Metadata, oldMetadata: Metadata): Boolean

    Compare the old metadata's schema with new metadata's schema for column mapping schema changes.

    Compare the old metadata's schema with new metadata's schema for column mapping schema changes. Also check for repartition because we need to fail fast when repartition detected.

    newMetadata's snapshot version must be >= oldMetadata's snapshot version so we could reliably detect the difference between ADD COLUMN and DROP COLUMN.

    As of now, newMetadata is column mapping read compatible with oldMetadata if no rename column or drop column has happened in-between.

32.  def hasPhysicalName(field: StructField): Boolean
33.  def hashCode(): Int

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

34.  def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

    Attributes

    protected

    Definition Classes

    Logging

35.  def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Attributes

    protected

    Definition Classes

    Logging

36.  def isColumnMappingUpgrade(oldMode: DeltaColumnMappingMode, newMode: DeltaColumnMappingMode): Boolean
37.  def isDropColumnOperation(newSchema: StructType, currentSchema: StructType): Boolean
38.  def isDropColumnOperation(newMetadata: Metadata, currentMetadata: Metadata): Boolean

    Returns true if Column Mapping mode is enabled and the newMetadata's schema, when compared to the currentMetadata's schema, is indicative of a DROP COLUMN operation.

    Returns true if Column Mapping mode is enabled and the newMetadata's schema, when compared to the currentMetadata's schema, is indicative of a DROP COLUMN operation.

    We detect DROP COLUMNS by checking if any physical name in currentSchema is missing in newSchema.

39.  final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

40.  def isInternalField(field: StructField): Boolean
41.  def isRenameColumnOperation(newSchema: StructType, currentSchema: StructType): Boolean
42.  def isRenameColumnOperation(newMetadata: Metadata, currentMetadata: Metadata): Boolean

    Returns true if Column Mapping mode is enabled and the newMetadata's schema, when compared to the currentMetadata's schema, is indicative of a RENAME COLUMN operation.

    Returns true if Column Mapping mode is enabled and the newMetadata's schema, when compared to the currentMetadata's schema, is indicative of a RENAME COLUMN operation.

    We detect RENAME COLUMNS by checking if any two columns with the same physical name have different logical names

43.  def isTraceEnabled(): Boolean

    Attributes

    protected

    Definition Classes

    Logging

44.  def log: Logger

    Attributes

    protected

    Definition Classes

    Logging

45.  def logConsole(line: String): Unit

    Definition Classes

    DatabricksLogging

46.  def logDebug(msg: => String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

47.  def logDebug(msg: => String): Unit

    Attributes

    protected

    Definition Classes

    Logging

48.  def logError(msg: => String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

49.  def logError(msg: => String): Unit

    Attributes

    protected

    Definition Classes

    Logging

50.  def logInfo(msg: => String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

51.  def logInfo(msg: => String): Unit

    Attributes

    protected

    Definition Classes

    Logging

52.  def logName: String

    Attributes

    protected

    Definition Classes

    Logging

53.  def logTrace(msg: => String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

54.  def logTrace(msg: => String): Unit

    Attributes

    protected

    Definition Classes

    Logging

55.  def logWarning(msg: => String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

56.  def logWarning(msg: => String): Unit

    Attributes

    protected

    Definition Classes

    Logging

57.  final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

58.  final def notify(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

59.  final def notifyAll(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

60.  def recordDeltaEvent(deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty, data: AnyRef = null, path: Option[Path] = None): Unit

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    path

    Used to log the path of the delta table when deltaLog is null.

    Attributes

    protected

    Definition Classes

    DeltaLogging

61.  def recordDeltaOperation[A](deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: => A): A

    Used to report the duration as well as the success or failure of an operation on a deltaLog.

    Used to report the duration as well as the success or failure of an operation on a deltaLog.

    Attributes

    protected

    Definition Classes

    DeltaLogging

62.  def recordDeltaOperationForTablePath[A](tablePath: String, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: => A): A

    Used to report the duration as well as the success or failure of an operation on a tahoePath.

    Used to report the duration as well as the success or failure of an operation on a tahoePath.

    Attributes

    protected

    Definition Classes

    DeltaLogging

63.  def recordEvent(metric: MetricDefinition, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit

    Definition Classes

    DatabricksLogging

64.  def recordFrameProfile[T](group: String, name: String)(thunk: => T): T

    Attributes

    protected

    Definition Classes

    DeltaLogging

65.  def recordOperation[S](opType: OpType, opTarget: String = null, extraTags: Map[TagDefinition, String], isSynchronous: Boolean = true, alwaysRecordStats: Boolean = false, allowAuthTags: Boolean = false, killJvmIfStuck: Boolean = false, outputMetric: MetricDefinition = METRIC_OPERATION_DURATION, silent: Boolean = true)(thunk: => S): S

    Definition Classes

    DatabricksLogging

66.  def recordProductEvent(metric: MetricDefinition with CentralizableMetric, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit

    Definition Classes

    DatabricksLogging

67.  def recordProductUsage(metric: MetricDefinition with CentralizableMetric, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit

    Definition Classes

    DatabricksLogging

68.  def recordUsage(metric: MetricDefinition, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit

    Definition Classes

    DatabricksLogging

69.  def renameColumns(schema: StructType): StructType

    Recursively renames columns in the given schema with their physical schema.

70.  def satisfiesColumnMappingProtocol(protocol: Protocol): Boolean
71.  def setPhysicalNames(schema: StructType, fieldPathToPhysicalName: Map[Seq[String], String]): StructType

    Set physical name based on field path, skip if field path not found in the map

72.  val supportedModes: Set[DeltaColumnMappingMode]
73.  final def synchronized[T0](arg0: => T0): T0

    Definition Classes

    AnyRef

74.  def toString(): String

    Definition Classes

    AnyRef → Any

75.  def verifyAndUpdateMetadataChange(deltaLog: DeltaLog, oldProtocol: Protocol, oldMetadata: Metadata, newMetadata: Metadata, isCreatingNewTable: Boolean, isOverwriteSchema: Boolean): Metadata

    If the table is already on the column mapping protocol, we block:

    If the table is already on the column mapping protocol, we block:

    • changing column mapping config otherwise, we block
    • upgrading to the column mapping Protocol through configurations
76.  def verifyInternalProperties(one: Map[String, String], two: Map[String, String]): Boolean
77.  final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.InterruptedException])

78.  final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.InterruptedException])

79.  final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.InterruptedException]) @native()

80.  def withStatusCode[T](statusCode: String, defaultMessage: String, data: Map[String, Any] = Map.empty)(body: => T): T

    Report a log to indicate some command is running.

    Report a log to indicate some command is running.

    Definition Classes

    DeltaProgressReporter