Packages

class DeltaCatalog extends AbstractDeltaCatalog

A Spark catalog plugin for Delta Lake tables that implements the Spark DataSource V2 Catalog API.

To use this catalog, configure it in your Spark session: 


// Scala example
val spark = SparkSession
  .builder()
  .appName("...")
  .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
  .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
  .getOrCreate()


// Python example
spark = SparkSession
  .builder
  .appName("...")
  .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
  .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
  .getOrCreate()

Architecture and Delegation Logic

This class sits in the delta-spark (unified) module and provides a single entry point for Delta Lake catalog operations.

The unified module can access both implementations:

  • V1 connector: DeltaTableV2 - Legacy connector using DeltaLog, full read/write support
  • V2 connector: SparkTable - sparkV2 connector, read-only support

See DeltaV2Mode for V1 vs V2 connector definitions and enable mode configuration.

Linear Supertypes
AbstractDeltaCatalog, DeltaLogging, DatabricksLogging, DeltaProgressReporter, Logging, SupportsPathIdentifier, StagingTableCatalog, DelegatingCatalogExtension, CatalogExtension, SupportsNamespaces, FunctionCatalog, TableCatalog, CatalogPlugin, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DeltaCatalog
  2. AbstractDeltaCatalog
  3. DeltaLogging
  4. DatabricksLogging
  5. DeltaProgressReporter
  6. Logging
  7. SupportsPathIdentifier
  8. StagingTableCatalog
  9. DelegatingCatalogExtension
  10. CatalogExtension
  11. SupportsNamespaces
  12. FunctionCatalog
  13. TableCatalog
  14. CatalogPlugin
  15. AnyRef
  16. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new DeltaCatalog()

Type Members

  1. implicit class LogStringContext extends AnyRef
    Definition Classes
    Logging

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def alterNamespace(arg0: Array[String], arg1: <repeated...>[NamespaceChange]): Unit
    Definition Classes
    DelegatingCatalogExtension → SupportsNamespaces
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException]) @transient()
  5. def alterTable(ident: Identifier, changes: TableChange*): Table
    Definition Classes
    AbstractDeltaCatalog
  6. def alterTable(arg0: Identifier, arg1: <repeated...>[TableChange]): Table
    Definition Classes
    DelegatingCatalogExtension → TableCatalog
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchTableException]) @transient()
  7. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  8. def capabilities(): Set[TableCatalogCapability]
    Definition Classes
    DelegatingCatalogExtension → TableCatalog
  9. lazy val catalog: SessionCatalog
    Attributes
    protected
    Definition Classes
    SupportsPathIdentifier
  10. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  11. def createNamespace(arg0: Array[String], arg1: Map[String, String]): Unit
    Definition Classes
    DelegatingCatalogExtension → SupportsNamespaces
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NamespaceAlreadyExistsException])
  12. def createTable(ident: Identifier, schema: StructType, partitions: Array[Transform], properties: Map[String, String]): Table
    Definition Classes
    AbstractDeltaCatalog → DelegatingCatalogExtension → TableCatalog
  13. def createTable(ident: Identifier, columns: Array[connector.catalog.Column], partitions: Array[Transform], properties: Map[String, String]): Table
    Definition Classes
    AbstractDeltaCatalog → DelegatingCatalogExtension → TableCatalog
  14. def defaultNamespace(): Array[String]
    Definition Classes
    DelegatingCatalogExtension → CatalogPlugin
  15. def deltaAssert(check: => Boolean, name: String, msg: String, deltaLog: DeltaLog, data: AnyRef, path: Option[Path]): Unit
    Attributes
    protected
    Definition Classes
    DeltaLogging
  16. def dropNamespace(arg0: Array[String], arg1: Boolean): Boolean
    Definition Classes
    DelegatingCatalogExtension → SupportsNamespaces
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException]) @throws(classOf[org.apache.spark.sql.catalyst.analysis.NonEmptyNamespaceException])
  17. def dropTable(arg0: Identifier): Boolean
    Definition Classes
    DelegatingCatalogExtension → TableCatalog
  18. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  19. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  20. def functionExists(arg0: Identifier): Boolean
    Definition Classes
    DelegatingCatalogExtension → FunctionCatalog
  21. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  22. def getCommonTags(deltaLog: DeltaLog, tahoeId: String): Map[TagDefinition, String]
    Definition Classes
    DeltaLogging
  23. def getErrorData(e: Throwable): Map[String, Any]
    Definition Classes
    DeltaLogging
  24. def getExistingTableIfExists(table: TableIdentifier, identOpt: Option[Identifier], operation: CreationMode): Option[CatalogTable]
    Definition Classes
    AbstractDeltaCatalog
  25. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  26. def initialize(arg0: String, arg1: CaseInsensitiveStringMap): Unit
    Definition Classes
    DelegatingCatalogExtension → CatalogPlugin
  27. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  28. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  29. def invalidateTable(arg0: Identifier): Unit
    Definition Classes
    DelegatingCatalogExtension → TableCatalog
  30. def isIcebergPathIdentifier(ident: Identifier): Boolean
    Attributes
    protected
    Definition Classes
    SupportsPathIdentifier
  31. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  32. def isPathIdentifier(tableIdentifier: TableIdentifier): Boolean
    Attributes
    protected
    Definition Classes
    SupportsPathIdentifier
  33. def isPathIdentifier(table: CatalogTable): Boolean
    Attributes
    protected
    Definition Classes
    SupportsPathIdentifier
  34. def isPathIdentifier(ident: Identifier): Boolean
    Attributes
    protected
    Definition Classes
    SupportsPathIdentifier
  35. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  36. def listFunctions(arg0: Array[String]): Array[Identifier]
    Definition Classes
    DelegatingCatalogExtension → FunctionCatalog
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException])
  37. def listNamespaces(arg0: Array[String]): Array[Array[String]]
    Definition Classes
    DelegatingCatalogExtension → SupportsNamespaces
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException])
  38. def listNamespaces(): Array[Array[String]]
    Definition Classes
    DelegatingCatalogExtension → SupportsNamespaces
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException])
  39. def listTables(arg0: Array[String]): Array[Identifier]
    Definition Classes
    DelegatingCatalogExtension → TableCatalog
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException])
  40. def loadCatalogTable(ident: Identifier, catalogTable: CatalogTable): Table

    Loads a Delta table that is registered in the catalog.

    Loads a Delta table that is registered in the catalog.

    Routing logic based on DeltaV2Mode:

    • STRICT: Returns sparkV2 SparkTable (V2 connector)
    • NONE (default): Returns DeltaTableV2 (V1 connector)
    ident

    The identifier of the table in the catalog.

    catalogTable

    The catalog table metadata containing table properties and location.

    returns

    Table instance (SparkTable for V2, DeltaTableV2 for V1).

    Definition Classes
    DeltaCatalog → AbstractDeltaCatalog
    Annotations
    @Override()
  41. def loadFunction(arg0: Identifier): UnboundFunction
    Definition Classes
    DelegatingCatalogExtension → FunctionCatalog
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchFunctionException])
  42. def loadNamespaceMetadata(arg0: Array[String]): Map[String, String]
    Definition Classes
    DelegatingCatalogExtension → SupportsNamespaces
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException])
  43. def loadPathTable(ident: Identifier): Table

    Loads a Delta table directly from a path.

    Loads a Delta table directly from a path. This is used for path-based table access where the identifier name is the table path.

    Routing logic based on DeltaV2Mode:

    • STRICT: Returns sparkV2 SparkTable (V2 connector)
    • NONE (default): Returns DeltaTableV2 (V1 connector)
    ident

    The identifier whose name contains the path to the Delta table.

    returns

    Table instance (SparkTable for V2, DeltaTableV2 for V1).

    Definition Classes
    DeltaCatalog → AbstractDeltaCatalog
    Annotations
    @Override()
  44. def loadTable(ident: Identifier, version: String): Table
    Definition Classes
    AbstractDeltaCatalog → DelegatingCatalogExtension → TableCatalog
  45. def loadTable(ident: Identifier, timestamp: Long): Table
    Definition Classes
    AbstractDeltaCatalog → DelegatingCatalogExtension → TableCatalog
  46. def loadTable(ident: Identifier): Table
    Definition Classes
    AbstractDeltaCatalog → DelegatingCatalogExtension → TableCatalog
  47. def loadTable(arg0: Identifier, arg1: Set[TableWritePrivilege]): Table
    Definition Classes
    TableCatalog
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchTableException])
  48. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  49. def logConsole(line: String): Unit
    Definition Classes
    DatabricksLogging
  50. def logDebug(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  51. def logDebug(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  52. def logDebug(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  53. def logDebug(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  54. def logError(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  55. def logError(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  56. def logError(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  57. def logError(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  58. def logInfo(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  59. def logInfo(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  60. def logInfo(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  61. def logInfo(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  62. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  63. def logTrace(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  64. def logTrace(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  65. def logTrace(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  66. def logTrace(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  67. def logWarning(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  68. def logWarning(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  69. def logWarning(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  70. def logWarning(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  71. def name(): String
    Definition Classes
    DelegatingCatalogExtension → CatalogPlugin
  72. def namespaceExists(arg0: Array[String]): Boolean
    Definition Classes
    DelegatingCatalogExtension → SupportsNamespaces
  73. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  74. def newIcebergPathTable(ident: Identifier): IcebergTablePlaceHolder
    Attributes
    protected
    Definition Classes
    SupportsPathIdentifier
  75. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  76. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  77. def purgeTable(arg0: Identifier): Boolean
    Definition Classes
    DelegatingCatalogExtension → TableCatalog
  78. def recordDeltaEvent(deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String], data: AnyRef, path: Option[Path]): Unit
    Attributes
    protected
    Definition Classes
    DeltaLogging
  79. def recordDeltaOperation[A](deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String])(thunk: => A): A
    Attributes
    protected
    Definition Classes
    DeltaLogging
  80. def recordDeltaOperationForTablePath[A](tablePath: String, opType: String, tags: Map[TagDefinition, String])(thunk: => A): A
    Attributes
    protected
    Definition Classes
    DeltaLogging
  81. def recordEvent(metric: MetricDefinition, additionalTags: Map[TagDefinition, String], blob: String, trimBlob: Boolean): Unit
    Definition Classes
    DatabricksLogging
  82. def recordFrameProfile[T](group: String, name: String)(thunk: => T): T
    Attributes
    protected
    Definition Classes
    DeltaLogging
  83. def recordOperation[S](opType: OpType, opTarget: String, extraTags: Map[TagDefinition, String], isSynchronous: Boolean, alwaysRecordStats: Boolean, allowAuthTags: Boolean, killJvmIfStuck: Boolean, outputMetric: MetricDefinition, silent: Boolean)(thunk: => S): S
    Definition Classes
    DatabricksLogging
  84. def recordProductEvent(metric: MetricDefinition with CentralizableMetric, additionalTags: Map[TagDefinition, String], blob: String, trimBlob: Boolean): Unit
    Definition Classes
    DatabricksLogging
  85. def recordProductUsage(metric: MetricDefinition with CentralizableMetric, quantity: Double, additionalTags: Map[TagDefinition, String], blob: String, forceSample: Boolean, trimBlob: Boolean, silent: Boolean): Unit
    Definition Classes
    DatabricksLogging
  86. def recordUsage(metric: MetricDefinition, quantity: Double, additionalTags: Map[TagDefinition, String], blob: String, forceSample: Boolean, trimBlob: Boolean, silent: Boolean): Unit
    Definition Classes
    DatabricksLogging
  87. def renameTable(arg0: Identifier, arg1: Identifier): Unit
    Definition Classes
    DelegatingCatalogExtension → TableCatalog
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchTableException]) @throws(classOf[org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException])
  88. final def setDelegateCatalog(arg0: CatalogPlugin): Unit
    Definition Classes
    DelegatingCatalogExtension → CatalogExtension
  89. val spark: Session
    Definition Classes
    AbstractDeltaCatalog
  90. def stageCreate(ident: Identifier, schema: StructType, partitions: Array[Transform], properties: Map[String, String]): StagedTable
    Definition Classes
    AbstractDeltaCatalog → StagingTableCatalog
  91. def stageCreate(arg0: Identifier, arg1: Array[connector.catalog.Column], arg2: Array[Transform], arg3: Map[String, String]): StagedTable
    Definition Classes
    StagingTableCatalog
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException]) @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException])
  92. def stageCreateOrReplace(ident: Identifier, schema: StructType, partitions: Array[Transform], properties: Map[String, String]): StagedTable
    Definition Classes
    AbstractDeltaCatalog → StagingTableCatalog
  93. def stageCreateOrReplace(arg0: Identifier, arg1: Array[connector.catalog.Column], arg2: Array[Transform], arg3: Map[String, String]): StagedTable
    Definition Classes
    StagingTableCatalog
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException])
  94. def stageReplace(ident: Identifier, schema: StructType, partitions: Array[Transform], properties: Map[String, String]): StagedTable
    Definition Classes
    AbstractDeltaCatalog → StagingTableCatalog
  95. def stageReplace(arg0: Identifier, arg1: Array[connector.catalog.Column], arg2: Array[Transform], arg3: Map[String, String]): StagedTable
    Definition Classes
    StagingTableCatalog
    Annotations
    @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException]) @throws(classOf[org.apache.spark.sql.catalyst.analysis.NoSuchTableException])
  96. def supportedCustomMetrics(): Array[CustomMetric]
    Definition Classes
    StagingTableCatalog
  97. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  98. def tableExists(ident: Identifier): Boolean
    Definition Classes
    SupportsPathIdentifier → TableCatalog
  99. def toString(): String
    Definition Classes
    AnyRef → Any
  100. def useNullableQuerySchema(): Boolean
    Definition Classes
    TableCatalog
  101. def validateClusterBySpec(maybeClusterBySpec: Option[ClusterBySpec], schema: StructType): Unit
    Definition Classes
    AbstractDeltaCatalog
  102. def verifyTableAndSolidify(tableDesc: CatalogTable, query: Option[LogicalPlan], maybeClusterBySpec: Option[ClusterBySpec]): CatalogTable
    Definition Classes
    AbstractDeltaCatalog
  103. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  104. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  105. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  106. def withLogContext(context: Map[String, String])(body: => Unit): Unit
    Attributes
    protected
    Definition Classes
    Logging
  107. def withStatusCode[T](statusCode: String, defaultMessage: String, data: Map[String, Any])(body: => T): T
    Definition Classes
    DeltaProgressReporter

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from AbstractDeltaCatalog

Inherited from DeltaLogging

Inherited from DatabricksLogging

Inherited from DeltaProgressReporter

Inherited from Logging

Inherited from SupportsPathIdentifier

Inherited from StagingTableCatalog

Inherited from DelegatingCatalogExtension

Inherited from CatalogExtension

Inherited from SupportsNamespaces

Inherited from FunctionCatalog

Inherited from TableCatalog

Inherited from CatalogPlugin

Inherited from AnyRef

Inherited from Any

Ungrouped