Packages

class DeltaCatalog extends DelegatingCatalogExtension with StagingTableCatalog with SupportsPathIdentifier with DeltaLogging

A Catalog extension which can properly handle the interaction between the HiveMetaStore and Delta tables. It delegates all operations DataSources other than Delta to the SparkCatalog.

Linear Supertypes
DeltaLogging, DatabricksLogging, DeltaProgressReporter, LoggingShims, Logging, SupportsPathIdentifier, StagingTableCatalog, DelegatingCatalogExtension, CatalogExtension, SupportsNamespaces, FunctionCatalog, TableCatalog, CatalogPlugin, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DeltaCatalog
  2. DeltaLogging
  3. DatabricksLogging
  4. DeltaProgressReporter
  5. LoggingShims
  6. Logging
  7. SupportsPathIdentifier
  8. StagingTableCatalog
  9. DelegatingCatalogExtension
  10. CatalogExtension
  11. SupportsNamespaces
  12. FunctionCatalog
  13. TableCatalog
  14. CatalogPlugin
  15. AnyRef
  16. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new DeltaCatalog()

Type Members

  1. implicit class LogStringContext extends AnyRef
    Definition Classes
    LoggingShims

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def alterNamespace(arg0: Array[String], arg1: <repeated...>[NamespaceChange]): Unit
    Definition Classes
    DelegatingCatalogExtension → SupportsNamespaces
    Annotations
    @throws( ... ) @transient()
  5. def alterTable(ident: Identifier, changes: TableChange*): Table
  6. def alterTable(arg0: Identifier, arg1: <repeated...>[TableChange]): Table
    Definition Classes
    DelegatingCatalogExtension → TableCatalog
    Annotations
    @throws( ... ) @transient()
  7. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  8. def capabilities(): Set[TableCatalogCapability]
    Definition Classes
    DelegatingCatalogExtension → TableCatalog
  9. lazy val catalog: SessionCatalog
    Attributes
    protected
    Definition Classes
    SupportsPathIdentifier
  10. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @IntrinsicCandidate()
  11. def createNamespace(arg0: Array[String], arg1: Map[String, String]): Unit
    Definition Classes
    DelegatingCatalogExtension → SupportsNamespaces
    Annotations
    @throws( ... )
  12. def createTable(ident: Identifier, schema: StructType, partitions: Array[Transform], properties: Map[String, String]): Table
    Definition Classes
    DeltaCatalog → DelegatingCatalogExtension → TableCatalog
  13. def createTable(ident: Identifier, columns: Array[connector.catalog.Column], partitions: Array[Transform], properties: Map[String, String]): Table
    Definition Classes
    DeltaCatalog → DelegatingCatalogExtension → TableCatalog
  14. def defaultNamespace(): Array[String]
    Definition Classes
    DelegatingCatalogExtension → CatalogPlugin
  15. def deltaAssert(check: ⇒ Boolean, name: String, msg: String, deltaLog: DeltaLog = null, data: AnyRef = null, path: Option[Path] = None): Unit

    Helper method to check invariants in Delta code.

    Helper method to check invariants in Delta code. Fails when running in tests, records a delta assertion event and logs a warning otherwise.

    Attributes
    protected
    Definition Classes
    DeltaLogging
  16. def dropNamespace(arg0: Array[String], arg1: Boolean): Boolean
    Definition Classes
    DelegatingCatalogExtension → SupportsNamespaces
    Annotations
    @throws( ... ) @throws( ... )
  17. def dropTable(arg0: Identifier): Boolean
    Definition Classes
    DelegatingCatalogExtension → TableCatalog
  18. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  19. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  20. def functionExists(arg0: Identifier): Boolean
    Definition Classes
    DelegatingCatalogExtension → FunctionCatalog
  21. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @IntrinsicCandidate()
  22. def getCommonTags(deltaLog: DeltaLog, tahoeId: String): Map[TagDefinition, String]
    Definition Classes
    DeltaLogging
  23. def getErrorData(e: Throwable): Map[String, Any]
    Definition Classes
    DeltaLogging
  24. def getExistingTableIfExists(table: TableIdentifier): Option[CatalogTable]

    Checks if a table already exists for the provided identifier.

  25. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @IntrinsicCandidate()
  26. def initialize(arg0: String, arg1: CaseInsensitiveStringMap): Unit
    Definition Classes
    DelegatingCatalogExtension → CatalogPlugin
  27. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  28. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  29. def invalidateTable(arg0: Identifier): Unit
    Definition Classes
    DelegatingCatalogExtension → TableCatalog
  30. def isIcebergPathIdentifier(ident: Identifier): Boolean
    Attributes
    protected
    Definition Classes
    SupportsPathIdentifier
  31. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  32. def isPathIdentifier(tableIdentifier: TableIdentifier): Boolean
    Attributes
    protected
    Definition Classes
    SupportsPathIdentifier
  33. def isPathIdentifier(table: CatalogTable): Boolean
    Attributes
    protected
    Definition Classes
    SupportsPathIdentifier
  34. def isPathIdentifier(ident: Identifier): Boolean
    Attributes
    protected
    Definition Classes
    SupportsPathIdentifier
  35. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  36. def listFunctions(arg0: Array[String]): Array[Identifier]
    Definition Classes
    DelegatingCatalogExtension → FunctionCatalog
    Annotations
    @throws( ... )
  37. def listNamespaces(arg0: Array[String]): Array[Array[String]]
    Definition Classes
    DelegatingCatalogExtension → SupportsNamespaces
    Annotations
    @throws( ... )
  38. def listNamespaces(): Array[Array[String]]
    Definition Classes
    DelegatingCatalogExtension → SupportsNamespaces
    Annotations
    @throws( ... )
  39. def listTables(arg0: Array[String]): Array[Identifier]
    Definition Classes
    DelegatingCatalogExtension → TableCatalog
    Annotations
    @throws( ... )
  40. def loadFunction(arg0: Identifier): UnboundFunction
    Definition Classes
    DelegatingCatalogExtension → FunctionCatalog
    Annotations
    @throws( ... )
  41. def loadNamespaceMetadata(arg0: Array[String]): Map[String, String]
    Definition Classes
    DelegatingCatalogExtension → SupportsNamespaces
    Annotations
    @throws( ... )
  42. def loadTable(ident: Identifier, version: String): Table
    Definition Classes
    DeltaCatalog → DelegatingCatalogExtension → TableCatalog
  43. def loadTable(ident: Identifier, timestamp: Long): Table
    Definition Classes
    DeltaCatalog → DelegatingCatalogExtension → TableCatalog
  44. def loadTable(ident: Identifier): Table
    Definition Classes
    DeltaCatalog → DelegatingCatalogExtension → TableCatalog
  45. def loadTable(arg0: Identifier, arg1: Set[TableWritePrivilege]): Table
    Definition Classes
    TableCatalog
    Annotations
    @throws( ... )
  46. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  47. def logConsole(line: String): Unit
    Definition Classes
    DatabricksLogging
  48. def logDebug(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  49. def logDebug(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  50. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  51. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  52. def logError(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  53. def logError(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  54. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  55. def logError(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  56. def logInfo(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  57. def logInfo(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  58. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  59. def logInfo(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  60. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  61. def logTrace(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  62. def logTrace(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  63. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  64. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  65. def logWarning(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  66. def logWarning(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  67. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  68. def logWarning(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  69. def name(): String
    Definition Classes
    DelegatingCatalogExtension → CatalogPlugin
  70. def namespaceExists(arg0: Array[String]): Boolean
    Definition Classes
    DelegatingCatalogExtension → SupportsNamespaces
  71. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  72. def newDeltaPathTable(ident: Identifier): DeltaTableV2
    Attributes
    protected
  73. def newIcebergPathTable(ident: Identifier): IcebergTablePlaceHolder
    Attributes
    protected
    Definition Classes
    SupportsPathIdentifier
  74. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @IntrinsicCandidate()
  75. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @IntrinsicCandidate()
  76. def purgeTable(arg0: Identifier): Boolean
    Definition Classes
    DelegatingCatalogExtension → TableCatalog
  77. def recordDeltaEvent(deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty, data: AnyRef = null, path: Option[Path] = None): Unit

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    path

    Used to log the path of the delta table when deltaLog is null.

    Attributes
    protected
    Definition Classes
    DeltaLogging
  78. def recordDeltaOperation[A](deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: ⇒ A): A

    Used to report the duration as well as the success or failure of an operation on a deltaLog.

    Used to report the duration as well as the success or failure of an operation on a deltaLog.

    Attributes
    protected
    Definition Classes
    DeltaLogging
  79. def recordDeltaOperationForTablePath[A](tablePath: String, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: ⇒ A): A

    Used to report the duration as well as the success or failure of an operation on a tahoePath.

    Used to report the duration as well as the success or failure of an operation on a tahoePath.

    Attributes
    protected
    Definition Classes
    DeltaLogging
  80. def recordEvent(metric: MetricDefinition, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit
    Definition Classes
    DatabricksLogging
  81. def recordFrameProfile[T](group: String, name: String)(thunk: ⇒ T): T
    Attributes
    protected
    Definition Classes
    DeltaLogging
  82. def recordOperation[S](opType: OpType, opTarget: String = null, extraTags: Map[TagDefinition, String], isSynchronous: Boolean = true, alwaysRecordStats: Boolean = false, allowAuthTags: Boolean = false, killJvmIfStuck: Boolean = false, outputMetric: MetricDefinition = METRIC_OPERATION_DURATION, silent: Boolean = true)(thunk: ⇒ S): S
    Definition Classes
    DatabricksLogging
  83. def recordProductEvent(metric: MetricDefinition with CentralizableMetric, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit
    Definition Classes
    DatabricksLogging
  84. def recordProductUsage(metric: MetricDefinition with CentralizableMetric, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit
    Definition Classes
    DatabricksLogging
  85. def recordUsage(metric: MetricDefinition, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit
    Definition Classes
    DatabricksLogging
  86. def renameTable(arg0: Identifier, arg1: Identifier): Unit
    Definition Classes
    DelegatingCatalogExtension → TableCatalog
    Annotations
    @throws( ... ) @throws( ... )
  87. final def setDelegateCatalog(arg0: CatalogPlugin): Unit
    Definition Classes
    DelegatingCatalogExtension → CatalogExtension
  88. val spark: SparkSession
  89. def stageCreate(ident: Identifier, schema: StructType, partitions: Array[Transform], properties: Map[String, String]): StagedTable
    Definition Classes
    DeltaCatalog → StagingTableCatalog
  90. def stageCreate(arg0: Identifier, arg1: Array[connector.catalog.Column], arg2: Array[Transform], arg3: Map[String, String]): StagedTable
    Definition Classes
    StagingTableCatalog
    Annotations
    @throws( ... ) @throws( ... )
  91. def stageCreateOrReplace(ident: Identifier, schema: StructType, partitions: Array[Transform], properties: Map[String, String]): StagedTable
    Definition Classes
    DeltaCatalog → StagingTableCatalog
  92. def stageCreateOrReplace(arg0: Identifier, arg1: Array[connector.catalog.Column], arg2: Array[Transform], arg3: Map[String, String]): StagedTable
    Definition Classes
    StagingTableCatalog
    Annotations
    @throws( ... )
  93. def stageReplace(ident: Identifier, schema: StructType, partitions: Array[Transform], properties: Map[String, String]): StagedTable
    Definition Classes
    DeltaCatalog → StagingTableCatalog
  94. def stageReplace(arg0: Identifier, arg1: Array[connector.catalog.Column], arg2: Array[Transform], arg3: Map[String, String]): StagedTable
    Definition Classes
    StagingTableCatalog
    Annotations
    @throws( ... ) @throws( ... )
  95. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  96. def tableExists(ident: Identifier): Boolean
    Definition Classes
    SupportsPathIdentifier → TableCatalog
  97. def toString(): String
    Definition Classes
    AnyRef → Any
  98. def useNullableQuerySchema(): Boolean
    Definition Classes
    TableCatalog
  99. def validateClusterBySpec(maybeClusterBySpec: Option[ClusterBySpec], schema: StructType): Unit
  100. def verifyTableAndSolidify(tableDesc: CatalogTable, query: Option[LogicalPlan], maybeClusterBySpec: Option[ClusterBySpec] = None): CatalogTable

    Performs checks on the parameters provided for table creation for a Delta table.

  101. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  102. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  103. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  104. def withStatusCode[T](statusCode: String, defaultMessage: String, data: Map[String, Any] = Map.empty)(body: ⇒ T): T

    Report a log to indicate some command is running.

    Report a log to indicate some command is running.

    Definition Classes
    DeltaProgressReporter

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated
    Deprecated

Inherited from DeltaLogging

Inherited from DatabricksLogging

Inherited from DeltaProgressReporter

Inherited from LoggingShims

Inherited from Logging

Inherited from SupportsPathIdentifier

Inherited from StagingTableCatalog

Inherited from DelegatingCatalogExtension

Inherited from CatalogExtension

Inherited from SupportsNamespaces

Inherited from FunctionCatalog

Inherited from TableCatalog

Inherited from CatalogPlugin

Inherited from AnyRef

Inherited from Any

Ungrouped