Packages

class DeltaCatalog extends AbstractDeltaCatalog

A Spark catalog plugin for Delta Lake tables that implements the Spark DataSource V2 Catalog API.

To use this catalog, configure it in your Spark session: 


// Scala example
val spark = SparkSession
  .builder()
  .appName("...")
  .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
  .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
  .getOrCreate()


// Python example
spark = SparkSession
  .builder
  .appName("...")
  .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
  .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
  .getOrCreate()

Architecture and Delegation Logic

This class sits in the delta-spark (unified) module and provides a single entry point for Delta Lake catalog operations.

The unified module can access both implementations:

  • V1 connector: DeltaTableV2 - Legacy connector using DeltaLog, full read/write support
  • V2 connector: SparkTable - sparkV2 connector, read-only support

See DeltaV2Mode for V1 vs V2 connector definitions and enable mode configuration.

Linear Supertypes
AbstractDeltaCatalog, DeltaLogging, DatabricksLogging, DeltaProgressReporter, Logging, SupportsPathIdentifier, StagingTableCatalog, DelegatingCatalogExtension, CatalogExtension, SupportsNamespaces, FunctionCatalog, TableCatalog, CatalogPlugin, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DeltaCatalog
  2. AbstractDeltaCatalog
  3. DeltaLogging
  4. DatabricksLogging
  5. DeltaProgressReporter
  6. Logging
  7. SupportsPathIdentifier
  8. StagingTableCatalog
  9. DelegatingCatalogExtension
  10. CatalogExtension
  11. SupportsNamespaces
  12. FunctionCatalog
  13. TableCatalog
  14. CatalogPlugin
  15. AnyRef
  16. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new DeltaCatalog()

Type Members

  1. implicit class LogStringContext extends AnyRef
    Definition Classes
    Logging

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def MDC(key: LogKey, value: Any): MDC
    Attributes
    protected
    Definition Classes
    Logging
  5. def alterNamespace(arg0: Array[String], arg1: <repeated...>[NamespaceChange]): Unit
    Definition Classes
    DelegatingCatalogExtension → SupportsNamespaces
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException]) @transient()
  6. def alterTable(ident: Identifier, changes: TableChange*): Table
    Definition Classes
    AbstractDeltaCatalog
  7. def alterTable(arg0: Identifier, arg1: <repeated...>[TableChange]): Table
    Definition Classes
    DelegatingCatalogExtension → TableCatalog
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchTableException]) @transient()
  8. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  9. def capabilities(): Set[TableCatalogCapability]
    Definition Classes
    DelegatingCatalogExtension → TableCatalog
  10. lazy val catalog: SessionCatalog
    Attributes
    protected
    Definition Classes
    SupportsPathIdentifier
  11. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  12. def createNamespace(arg0: Array[String], arg1: Map[String, String]): Unit
    Definition Classes
    DelegatingCatalogExtension → SupportsNamespaces
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NamespaceAlreadyExistsException])
  13. def createTable(ident: Identifier, schema: StructType, partitions: Array[Transform], properties: Map[String, String]): Table
    Definition Classes
    AbstractDeltaCatalog → DelegatingCatalogExtension → TableCatalog
  14. def createTable(ident: Identifier, columns: Array[connector.catalog.Column], partitions: Array[Transform], properties: Map[String, String]): Table
    Definition Classes
    AbstractDeltaCatalog → DelegatingCatalogExtension → TableCatalog
  15. def createTable(arg0: Identifier, arg1: TableInfo): Table
    Definition Classes
    TableCatalog
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException]) @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException])
  16. def defaultNamespace(): Array[String]
    Definition Classes
    DelegatingCatalogExtension → CatalogPlugin
  17. def deltaAssert(check: => Boolean, name: String, msg: String, deltaLog: DeltaLog, data: AnyRef, path: Option[Path]): Unit
    Attributes
    protected
    Definition Classes
    DeltaLogging
  18. def dropNamespace(arg0: Array[String], arg1: Boolean): Boolean
    Definition Classes
    DelegatingCatalogExtension → SupportsNamespaces
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException]) @throws(classOf[org.apache.spark.sql.catalyst.analysis.NonEmptyNamespaceException])
  19. def dropTable(arg0: Identifier): Boolean
    Definition Classes
    DelegatingCatalogExtension → TableCatalog
  20. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  21. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  22. def functionExists(arg0: Identifier): Boolean
    Definition Classes
    DelegatingCatalogExtension → FunctionCatalog
  23. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  24. def getCommonTags(deltaLog: DeltaLog, tahoeId: String): Map[TagDefinition, String]
    Definition Classes
    DeltaLogging
  25. def getErrorData(e: Throwable): Map[String, Any]
    Definition Classes
    DeltaLogging
  26. def getExistingTableIfExists(table: TableIdentifier): Option[CatalogTable]
    Definition Classes
    AbstractDeltaCatalog
  27. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  28. def initialize(arg0: String, arg1: CaseInsensitiveStringMap): Unit
    Definition Classes
    DelegatingCatalogExtension → CatalogPlugin
  29. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  30. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  31. def invalidateTable(arg0: Identifier): Unit
    Definition Classes
    DelegatingCatalogExtension → TableCatalog
  32. def isIcebergPathIdentifier(ident: Identifier): Boolean
    Attributes
    protected
    Definition Classes
    SupportsPathIdentifier
  33. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  34. def isPathIdentifier(tableIdentifier: TableIdentifier): Boolean
    Attributes
    protected
    Definition Classes
    SupportsPathIdentifier
  35. def isPathIdentifier(table: CatalogTable): Boolean
    Attributes
    protected
    Definition Classes
    SupportsPathIdentifier
  36. def isPathIdentifier(ident: Identifier): Boolean
    Attributes
    protected
    Definition Classes
    SupportsPathIdentifier
  37. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  38. def listFunctions(arg0: Array[String]): Array[Identifier]
    Definition Classes
    DelegatingCatalogExtension → FunctionCatalog
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException])
  39. def listNamespaces(arg0: Array[String]): Array[Array[String]]
    Definition Classes
    DelegatingCatalogExtension → SupportsNamespaces
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException])
  40. def listNamespaces(): Array[Array[String]]
    Definition Classes
    DelegatingCatalogExtension → SupportsNamespaces
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException])
  41. def listTableSummaries(arg0: Array[String]): Array[TableSummary]
    Definition Classes
    TableCatalog
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException]) @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchTableException])
  42. def listTables(arg0: Array[String]): Array[Identifier]
    Definition Classes
    DelegatingCatalogExtension → TableCatalog
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException])
  43. def loadCatalogTable(ident: Identifier, catalogTable: CatalogTable): Table

    Loads a Delta table that is registered in the catalog.

    Loads a Delta table that is registered in the catalog.

    Routing logic based on DeltaV2Mode:

    • STRICT: Returns sparkV2 SparkTable (V2 connector)
    • NONE (default): Returns DeltaTableV2 (V1 connector)
    ident

    The identifier of the table in the catalog.

    catalogTable

    The catalog table metadata containing table properties and location.

    returns

    Table instance (SparkTable for V2, DeltaTableV2 for V1).

    Definition Classes
    DeltaCatalog → AbstractDeltaCatalog
    Annotations
    @Override()
  44. def loadFunction(arg0: Identifier): UnboundFunction
    Definition Classes
    DelegatingCatalogExtension → FunctionCatalog
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchFunctionException])
  45. def loadNamespaceMetadata(arg0: Array[String]): Map[String, String]
    Definition Classes
    DelegatingCatalogExtension → SupportsNamespaces
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException])
  46. def loadPathTable(ident: Identifier): Table

    Loads a Delta table directly from a path.

    Loads a Delta table directly from a path. This is used for path-based table access where the identifier name is the table path.

    Routing logic based on DeltaV2Mode:

    • STRICT: Returns sparkV2 SparkTable (V2 connector)
    • NONE (default): Returns DeltaTableV2 (V1 connector)
    ident

    The identifier whose name contains the path to the Delta table.

    returns

    Table instance (SparkTable for V2, DeltaTableV2 for V1).

    Definition Classes
    DeltaCatalog → AbstractDeltaCatalog
    Annotations
    @Override()
  47. def loadTable(ident: Identifier, version: String): Table
    Definition Classes
    AbstractDeltaCatalog → DelegatingCatalogExtension → TableCatalog
  48. def loadTable(ident: Identifier, timestamp: Long): Table
    Definition Classes
    AbstractDeltaCatalog → DelegatingCatalogExtension → TableCatalog
  49. def loadTable(ident: Identifier): Table
    Definition Classes
    AbstractDeltaCatalog → DelegatingCatalogExtension → TableCatalog
  50. def loadTable(arg0: Identifier, arg1: Set[TableWritePrivilege]): Table
    Definition Classes
    TableCatalog
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchTableException])
  51. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  52. def logBasedOnLevel(level: Level)(f: => MessageWithContext): Unit
    Attributes
    protected
    Definition Classes
    Logging
  53. def logConsole(line: String): Unit
    Definition Classes
    DatabricksLogging
  54. def logDebug(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  55. def logDebug(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  56. def logDebug(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  57. def logDebug(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  58. def logError(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  59. def logError(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  60. def logError(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  61. def logError(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  62. def logInfo(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  63. def logInfo(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  64. def logInfo(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  65. def logInfo(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  66. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  67. def logTrace(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  68. def logTrace(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  69. def logTrace(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  70. def logTrace(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  71. def logWarning(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  72. def logWarning(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  73. def logWarning(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  74. def logWarning(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  75. def name(): String
    Definition Classes
    DelegatingCatalogExtension → CatalogPlugin
  76. def namespaceExists(arg0: Array[String]): Boolean
    Definition Classes
    DelegatingCatalogExtension → SupportsNamespaces
  77. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  78. def newIcebergPathTable(ident: Identifier): IcebergTablePlaceHolder
    Attributes
    protected
    Definition Classes
    SupportsPathIdentifier
  79. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  80. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  81. def purgeTable(arg0: Identifier): Boolean
    Definition Classes
    DelegatingCatalogExtension → TableCatalog
  82. def recordDeltaEvent(deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String], data: AnyRef, path: Option[Path]): Unit
    Attributes
    protected
    Definition Classes
    DeltaLogging
  83. def recordDeltaOperation[A](deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String])(thunk: => A): A
    Attributes
    protected
    Definition Classes
    DeltaLogging
  84. def recordDeltaOperationForTablePath[A](tablePath: String, opType: String, tags: Map[TagDefinition, String])(thunk: => A): A
    Attributes
    protected
    Definition Classes
    DeltaLogging
  85. def recordEvent(metric: MetricDefinition, additionalTags: Map[TagDefinition, String], blob: String, trimBlob: Boolean): Unit
    Definition Classes
    DatabricksLogging
  86. def recordFrameProfile[T](group: String, name: String)(thunk: => T): T
    Attributes
    protected
    Definition Classes
    DeltaLogging
  87. def recordOperation[S](opType: OpType, opTarget: String, extraTags: Map[TagDefinition, String], isSynchronous: Boolean, alwaysRecordStats: Boolean, allowAuthTags: Boolean, killJvmIfStuck: Boolean, outputMetric: MetricDefinition, silent: Boolean)(thunk: => S): S
    Definition Classes
    DatabricksLogging
  88. def recordProductEvent(metric: MetricDefinition with CentralizableMetric, additionalTags: Map[TagDefinition, String], blob: String, trimBlob: Boolean): Unit
    Definition Classes
    DatabricksLogging
  89. def recordProductUsage(metric: MetricDefinition with CentralizableMetric, quantity: Double, additionalTags: Map[TagDefinition, String], blob: String, forceSample: Boolean, trimBlob: Boolean, silent: Boolean): Unit
    Definition Classes
    DatabricksLogging
  90. def recordUsage(metric: MetricDefinition, quantity: Double, additionalTags: Map[TagDefinition, String], blob: String, forceSample: Boolean, trimBlob: Boolean, silent: Boolean): Unit
    Definition Classes
    DatabricksLogging
  91. def renameTable(arg0: Identifier, arg1: Identifier): Unit
    Definition Classes
    DelegatingCatalogExtension → TableCatalog
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchTableException]) @throws(classOf[org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException])
  92. final def setDelegateCatalog(arg0: CatalogPlugin): Unit
    Definition Classes
    DelegatingCatalogExtension → CatalogExtension
  93. val spark: Session
    Definition Classes
    AbstractDeltaCatalog
  94. def stageCreate(ident: Identifier, schema: StructType, partitions: Array[Transform], properties: Map[String, String]): StagedTable
    Definition Classes
    AbstractDeltaCatalog → StagingTableCatalog
  95. def stageCreate(arg0: Identifier, arg1: TableInfo): StagedTable
    Definition Classes
    StagingTableCatalog
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException]) @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException])
  96. def stageCreateOrReplace(ident: Identifier, schema: StructType, partitions: Array[Transform], properties: Map[String, String]): StagedTable
    Definition Classes
    AbstractDeltaCatalog → StagingTableCatalog
  97. def stageCreateOrReplace(arg0: Identifier, arg1: TableInfo): StagedTable
    Definition Classes
    StagingTableCatalog
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException])
  98. def stageReplace(ident: Identifier, schema: StructType, partitions: Array[Transform], properties: Map[String, String]): StagedTable
    Definition Classes
    AbstractDeltaCatalog → StagingTableCatalog
  99. def stageReplace(arg0: Identifier, arg1: TableInfo): StagedTable
    Definition Classes
    StagingTableCatalog
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException]) @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchTableException])
  100. def supportedCustomMetrics(): Array[CustomMetric]
    Definition Classes
    StagingTableCatalog
  101. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  102. def tableExists(ident: Identifier): Boolean
    Definition Classes
    SupportsPathIdentifier → TableCatalog
  103. def toString(): String
    Definition Classes
    AnyRef → Any
  104. def useNullableQuerySchema(): Boolean
    Definition Classes
    TableCatalog
  105. def validateClusterBySpec(maybeClusterBySpec: Option[ClusterBySpec], schema: StructType): Unit
    Definition Classes
    AbstractDeltaCatalog
  106. def verifyTableAndSolidify(tableDesc: CatalogTable, query: Option[LogicalPlan], maybeClusterBySpec: Option[ClusterBySpec]): CatalogTable
    Definition Classes
    AbstractDeltaCatalog
  107. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  108. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  109. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  110. def withLogContext(context: Map[String, String])(body: => Unit): Unit
    Attributes
    protected
    Definition Classes
    Logging
  111. def withStatusCode[T](statusCode: String, defaultMessage: String, data: Map[String, Any])(body: => T): T
    Definition Classes
    DeltaProgressReporter

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

  2. def stageCreate(arg0: Identifier, arg1: Array[connector.catalog.Column], arg2: Array[Transform], arg3: Map[String, String]): StagedTable
    Definition Classes
    StagingTableCatalog
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException]) @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException]) @Deprecated
    Deprecated

    (Since version 4.1.0)

  3. def stageCreateOrReplace(arg0: Identifier, arg1: Array[connector.catalog.Column], arg2: Array[Transform], arg3: Map[String, String]): StagedTable
    Definition Classes
    StagingTableCatalog
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException]) @Deprecated
    Deprecated

    (Since version 4.1.0)

  4. def stageReplace(arg0: Identifier, arg1: Array[connector.catalog.Column], arg2: Array[Transform], arg3: Map[String, String]): StagedTable
    Definition Classes
    StagingTableCatalog
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException]) @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchTableException]) @Deprecated
    Deprecated

    (Since version 4.1.0)

Inherited from AbstractDeltaCatalog

Inherited from DeltaLogging

Inherited from DatabricksLogging

Inherited from DeltaProgressReporter

Inherited from Logging

Inherited from SupportsPathIdentifier

Inherited from StagingTableCatalog

Inherited from DelegatingCatalogExtension

Inherited from CatalogExtension

Inherited from SupportsNamespaces

Inherited from FunctionCatalog

Inherited from TableCatalog

Inherited from CatalogPlugin

Inherited from AnyRef

Inherited from Any

Ungrouped