case class DeltaSource(spark: SparkSession, deltaLog: DeltaLog, options: DeltaOptions, snapshotAtSourceInit: SnapshotDescriptor, metadataPath: String, metadataTrackingLog: Option[DeltaSourceMetadataTrackingLog] = None, filters: Seq[Expression] = Nil) extends DeltaSourceBase with DeltaSourceCDCSupport with DeltaSourceMetadataEvolutionSupport with Product with Serializable

A streaming source for a Delta table.

When a new stream is started, delta starts by constructing a org.apache.spark.sql.delta.Snapshot at the current version of the table. This snapshot is broken up into batches until all existing data has been processed. Subsequent processing is done by tailing the change log looking for new data. This results in the streaming query returning the same answer as a batch query that had processed the entire dataset at any given point.

Linear Supertypes
Serializable, Product, Equals, DeltaSourceMetadataEvolutionSupport, DeltaSourceCDCSupport, DeltaSourceBase, DeltaLogging, DatabricksLogging, DeltaProgressReporter, Logging, SupportsTriggerAvailableNow, SupportsAdmissionControl, Source, SparkDataStream, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DeltaSource
  2. Serializable
  3. Product
  4. Equals
  5. DeltaSourceMetadataEvolutionSupport
  6. DeltaSourceCDCSupport
  7. DeltaSourceBase
  8. DeltaLogging
  9. DatabricksLogging
  10. DeltaProgressReporter
  11. Logging
  12. SupportsTriggerAvailableNow
  13. SupportsAdmissionControl
  14. Source
  15. SparkDataStream
  16. AnyRef
  17. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new DeltaSource(spark: SparkSession, deltaLog: DeltaLog, options: DeltaOptions, snapshotAtSourceInit: SnapshotDescriptor, metadataPath: String, metadataTrackingLog: Option[DeltaSourceMetadataTrackingLog] = None, filters: Seq[Expression] = Nil)

Type Members

  1. case class AdmissionLimits(maxFiles: Option[Int] = options.maxFilesPerTrigger, bytesToTake: Long = options.maxBytesPerTrigger.getOrElse(Long.MaxValue)) extends DeltaSourceAdmissionBase with Product with Serializable

    Class that helps controlling how much data should be processed by a single micro-batch.

  2. trait DeltaSourceAdmissionBase extends AnyRef
  3. class IndexedChangeFileSeq extends AnyRef

    This class represents an iterator of Change metadata(AddFile, RemoveFile, AddCDCFile) for a particular version.

    This class represents an iterator of Change metadata(AddFile, RemoveFile, AddCDCFile) for a particular version.

    Definition Classes
    DeltaSourceCDCSupport

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. lazy val allowUnsafeStreamingReadOnColumnMappingSchemaChanges: Boolean

    Flag that allows user to force enable unsafe streaming read on Delta table with column mapping enabled AND drop/rename actions.

    Flag that allows user to force enable unsafe streaming read on Delta table with column mapping enabled AND drop/rename actions.

    Attributes
    protected
    Definition Classes
    DeltaSourceBase
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def checkReadIncompatibleSchemaChangeOnStreamStartOnce(batchStartVersion: Long, batchEndVersionOpt: Option[Long] = None): Unit

    Check read-incompatible schema changes during stream (re)start so we could fail fast.

    Check read-incompatible schema changes during stream (re)start so we could fail fast.

    This only needs to be called ONCE in the life cycle of a stream, either at the very first latestOffset, or the very first getBatch to make sure we have detected an incompatible schema change. Typically, the verifyStreamHygiene that was called maybe good enough to detect these schema changes, there may be cases that wouldn't work, e.g. consider this sequence: 1. User starts a new stream @ startingVersion 1 2. latestOffset is called before getBatch() because there was no previous commits so getBatch won't be called as a recovery mechanism. Suppose there's a single rename/drop/nullability change S during computing next offset, S would look exactly the same as the latest schema so verifyStreamHygiene would not work. 3. latestOffset would return this new offset cross the schema boundary.

    If a schema log is already initialized, we don't have to run the initialization nor schema checks any more.

    batchStartVersion

    Start version we want to verify read compatibility against

    batchEndVersionOpt

    Optionally, if we are checking against an existing constructed batch during streaming initialization, we would also like to verify all schema changes in between as well before we can lazily initialize the schema log if needed.

    Attributes
    protected
    Definition Classes
    DeltaSourceBase
  7. def checkReadIncompatibleSchemaChanges(metadata: Metadata, version: Long, batchStartVersion: Long, batchEndVersionOpt: Option[Long] = None, validatedDuringStreamStart: Boolean = false): Unit

    Narrow waist to verify a metadata action for read-incompatible schema changes, specifically: 1.

    Narrow waist to verify a metadata action for read-incompatible schema changes, specifically: 1. Any column mapping related schema changes (rename / drop) columns 2. Standard read-compatibility changes including: a) No missing columns b) No data type changes c) No read-incompatible nullability changes If the check fails, we throw an exception to exit the stream. If lazy log initialization is required, we also run a one time scan to safely initialize the metadata tracking log upon any non-additive schema change failures.

    metadata

    Metadata that contains a potential schema change

    version

    Version for the metadata action

    validatedDuringStreamStart

    Whether this check is being done during stream start.

    Attributes
    protected
    Definition Classes
    DeltaSourceBase
  8. def cleanUpSnapshotResources(): Unit
    Attributes
    protected
    Definition Classes
    DeltaSourceBase
  9. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  10. def collectMetadataActions(startVersion: Long, endVersion: Long): Seq[(Long, Metadata)]
    Attributes
    protected
    Definition Classes
    DeltaSourceMetadataEvolutionSupport
  11. def collectProtocolActions(startVersion: Long, endVersion: Long): Seq[(Long, Protocol)]
    Attributes
    protected
    Definition Classes
    DeltaSourceMetadataEvolutionSupport
  12. def commit(end: Offset): Unit
    Definition Classes
    DeltaSource → Source
  13. def commit(end: Offset): Unit
    Definition Classes
    Source → SparkDataStream
  14. def createDataFrame(indexedFiles: Iterator[IndexedFile]): DataFrame

    Given an iterator of file actions, create a DataFrame representing the files added to a table Only AddFile actions will be used to create the DataFrame.

    Given an iterator of file actions, create a DataFrame representing the files added to a table Only AddFile actions will be used to create the DataFrame.

    indexedFiles

    actions iterator from which to generate the DataFrame.

    Attributes
    protected
    Definition Classes
    DeltaSourceBase
  15. def createDataFrameBetweenOffsets(startVersion: Long, startIndex: Long, isStartingVersion: Boolean, startOffsetOption: Option[DeltaSourceOffset], endOffset: DeltaSourceOffset): DataFrame

    Return the DataFrame between start and end offset.

    Return the DataFrame between start and end offset.

    Attributes
    protected
    Definition Classes
    DeltaSourceBase
  16. val deltaLog: DeltaLog
  17. def deserializeOffset(json: String): Offset
    Definition Classes
    Source → SparkDataStream
  18. val emptyDataFrame: DataFrame
    Attributes
    protected
    Definition Classes
    DeltaSourceBase
  19. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  20. val excludeRegex: Option[Regex]
    Attributes
    protected
  21. val filters: Seq[Expression]
  22. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  23. lazy val forceEnableStreamingReadOnReadIncompatibleSchemaChangesDuringStreamStart: Boolean

    Flag that allows user to disable the read-compatibility check during stream start which protects against an corner case in which verifyStreamHygiene could not detect.

    Flag that allows user to disable the read-compatibility check during stream start which protects against an corner case in which verifyStreamHygiene could not detect. This is a bug fix but yet a potential behavior change, so we add a flag to fallback.

    Attributes
    protected
    Definition Classes
    DeltaSourceBase
  24. lazy val forceEnableUnsafeReadOnNullabilityChange: Boolean

    Flag that allow user to fallback to the legacy behavior in which user can allow nullable=false schema to read nullable=true data, which is incorrect but a behavior change regardless.

    Flag that allow user to fallback to the legacy behavior in which user can allow nullable=false schema to read nullable=true data, which is incorrect but a behavior change regardless.

    Attributes
    protected
    Definition Classes
    DeltaSourceBase
  25. def getBatch(startOffsetOption: Option[Offset], end: Offset): DataFrame
    Definition Classes
    DeltaSource → Source
  26. def getCDCFileChangesAndCreateDataFrame(startVersion: Long, startIndex: Long, isStartingVersion: Boolean, endOffset: DeltaSourceOffset): DataFrame

    Get the changes from startVersion, startIndex to the end for CDC case.

    Get the changes from startVersion, startIndex to the end for CDC case. We need to call CDCReader to get the CDC DataFrame.

    startVersion

    - calculated starting version

    startIndex

    - calculated starting index

    isStartingVersion

    - whether the stream has to return the initial snapshot or not

    endOffset

    - Offset that signifies the end of the stream.

    returns

    the DataFrame containing the file changes (AddFile, RemoveFile, AddCDCFile)

    Attributes
    protected
    Definition Classes
    DeltaSourceCDCSupport
  27. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  28. def getCommonTags(deltaLog: DeltaLog, tahoeId: String): Map[TagDefinition, String]
    Definition Classes
    DeltaLogging
  29. def getDefaultReadLimit(): ReadLimit
    Definition Classes
    DeltaSource → SupportsAdmissionControl
  30. def getFileChanges(fromVersion: Long, fromIndex: Long, isStartingVersion: Boolean, endOffset: Option[DeltaSourceOffset] = None, verifyMetadataAction: Boolean = true): ClosableIterator[IndexedFile]

    Get the changes starting from (startVersion, startIndex).

    Get the changes starting from (startVersion, startIndex). The start point should not be included in the result.

    endOffset

    If defined, do not return changes beyond this offset. If not defined, we must be scanning the log to find the next offset.

    verifyMetadataAction

    If true, we will break the stream when we detect any read-incompatible metadata changes.

    Attributes
    protected
  31. def getFileChangesAndCreateDataFrame(startVersion: Long, startIndex: Long, isStartingVersion: Boolean, endOffset: DeltaSourceOffset): DataFrame

    get the changes from startVersion, startIndex to the end

    get the changes from startVersion, startIndex to the end

    startVersion

    - calculated starting version

    startIndex

    - calculated starting index

    isStartingVersion

    - whether the stream has to return the initial snapshot or not

    endOffset

    - Offset that signifies the end of the stream.

    Attributes
    protected
    Definition Classes
    DeltaSourceBase
  32. def getFileChangesForCDC(fromVersion: Long, fromIndex: Long, isStartingVersion: Boolean, limits: Option[AdmissionLimits], endOffset: Option[DeltaSourceOffset], verifyMetadataAction: Boolean = true): Iterator[(Long, Iterator[IndexedFile])]

    Get the changes starting from (fromVersion, fromIndex).

    Get the changes starting from (fromVersion, fromIndex). fromVersion is included. It returns an iterator of (log_version, fileActions)

    If verifyMetadataAction = true, we will break the stream when we detect any read-incompatible metadata changes.

    Attributes
    protected
    Definition Classes
    DeltaSourceCDCSupport
  33. def getFileChangesWithRateLimit(fromVersion: Long, fromIndex: Long, isStartingVersion: Boolean, limits: Option[AdmissionLimits] = Some(AdmissionLimits())): ClosableIterator[IndexedFile]
    Attributes
    protected
    Definition Classes
    DeltaSourceBase
  34. def getMetadataOrProtocolChangeIndexedFileIterator(metadataChangeOpt: Option[Metadata], protocolChangeOpt: Option[Protocol], version: Long): ClosableIterator[IndexedFile]

    If the current stream metadata is not equal to the metadata change in metadataChangeOpt, return a metadata change barrier IndexedFile.

    If the current stream metadata is not equal to the metadata change in metadataChangeOpt, return a metadata change barrier IndexedFile. Only returns something if trackingMetadataChangeis true.

    Attributes
    protected
    Definition Classes
    DeltaSourceMetadataEvolutionSupport
  35. def getNextOffsetFromPreviousOffset(previousOffset: DeltaSourceOffset, limits: Option[AdmissionLimits]): Option[DeltaSourceOffset]

    Return the next offset when previous offset exists.

    Return the next offset when previous offset exists.

    Attributes
    protected
    Definition Classes
    DeltaSourceBase
  36. def getNextOffsetFromPreviousOffsetIfPendingSchemaChange(previousOffset: DeltaSourceOffset): Option[DeltaSourceOffset]

    If the given previous Delta source offset is a schema change offset, returns the appropriate next offset.

    If the given previous Delta source offset is a schema change offset, returns the appropriate next offset. This should be called before trying any other means of determining the next offset. If this returns None, then there is no schema change, and the caller should determine the next offset in the normal way.

    Attributes
    protected
    Definition Classes
    DeltaSourceMetadataEvolutionSupport
  37. def getOffset: Option[Offset]
    Definition Classes
    DeltaSource → Source
  38. def getSnapshotAt(version: Long): Iterator[IndexedFile]

    This method computes the initial snapshot to read when Delta Source was initialized on a fresh stream.

    This method computes the initial snapshot to read when Delta Source was initialized on a fresh stream.

    Attributes
    protected
  39. def getSnapshotFromDeltaLog(version: Long): Snapshot

    Narrow-waist for generating snapshot from Delta Log within Delta Source

    Narrow-waist for generating snapshot from Delta Log within Delta Source

    Attributes
    protected
  40. def getStartingOffsetFromSpecificDeltaVersion(fromVersion: Long, isStartingVersion: Boolean, limits: Option[AdmissionLimits]): Option[DeltaSourceOffset]

    Returns the offset that starts from a specific delta table version.

    Returns the offset that starts from a specific delta table version. This function is called when starting a new stream query.

    fromVersion

    The version of the delta table to calculate the offset from.

    isStartingVersion

    Whether the delta version is for the initial snapshot or not.

    limits

    Indicates how much data can be processed by a micro batch.

    Attributes
    protected
    Definition Classes
    DeltaSourceBase
  41. lazy val getStartingVersion: Option[Long]

    Extracts whether users provided the option to time travel a relation.

    Extracts whether users provided the option to time travel a relation. If a query restarts from a checkpoint and the checkpoint has recorded the offset, this method should never been called.

    Attributes
    protected
  42. val hasCheckedReadIncompatibleSchemaChangesOnStreamStart: Boolean

    A global flag to mark whether we have done a per-stream start check for column mapping schema changes (rename / drop).

    A global flag to mark whether we have done a per-stream start check for column mapping schema changes (rename / drop).

    Attributes
    protected
    Definition Classes
    DeltaSourceBase
    Annotations
    @volatile()
  43. def initForTriggerAvailableNowIfNeeded(startOffsetOpt: Option[DeltaSourceOffset]): Unit

    initialize the internal states for AvailableNow if this method is called first time after prepareForTriggerAvailableNow.

    initialize the internal states for AvailableNow if this method is called first time after prepareForTriggerAvailableNow.

    Attributes
    protected
    Definition Classes
    DeltaSourceBase
  44. def initLastOffsetForTriggerAvailableNow(startOffsetOpt: Option[DeltaSourceOffset]): Unit
    Attributes
    protected
    Definition Classes
    DeltaSourceBase
  45. def initialOffset(): Offset
    Definition Classes
    Source → SparkDataStream
  46. var initialState: DeltaSourceSnapshot
    Attributes
    protected
  47. var initialStateVersion: Long
    Attributes
    protected
  48. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  49. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  50. def initializeMetadataTrackingAndExitStream(batchStartVersion: Long, batchEndVersionOpt: Option[Long] = None, alwaysFailUponLogInitialized: Boolean = false): Unit

    Initialize the schema tracking log if an empty schema tracking log is provided.

    Initialize the schema tracking log if an empty schema tracking log is provided. This method also checks the range between batchStartVersion and batchEndVersion to ensure we a safe schema to be initialized in the log.

    batchStartVersion

    Start version of the batch of data to be proceed, it should typically be the schema that is safe to process incoming data.

    batchEndVersionOpt

    Optionally, if we are looking at a constructed batch with existing end offset, we need to double verify to ensure no read-incompatible within the batch range.

    alwaysFailUponLogInitialized

    Whether we should always fail with the schema evolution exception.

    Attributes
    protected
    Definition Classes
    DeltaSourceMetadataEvolutionSupport
  51. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  52. val isStreamingFromColumnMappingTable: Boolean

    Whether we are streaming from a table with column mapping enabled

    Whether we are streaming from a table with column mapping enabled

    Attributes
    protected
    Definition Classes
    DeltaSourceBase
  53. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  54. val lastOffsetForTriggerAvailableNow: Option[DeltaSourceOffset]

    When AvailableNow is used, this offset will be the upper bound where this run of the query will process up.

    When AvailableNow is used, this offset will be the upper bound where this run of the query will process up. We may run multiple micro batches, but the query will stop itself when it reaches this offset.

    Attributes
    protected
    Definition Classes
    DeltaSourceBase
  55. def latestOffset(startOffset: Offset, limit: ReadLimit): Offset

    This should only be called by the engine.

    This should only be called by the engine. Call latestOffsetInternal instead if you need to get the latest offset.

    Definition Classes
    DeltaSource → SupportsAdmissionControl
  56. def latestOffsetInternal(startOffset: Option[DeltaSourceOffset], limit: ReadLimit): Option[DeltaSourceOffset]

    An internal latestOffsetInternal to get the latest offset.

    An internal latestOffsetInternal to get the latest offset.

    Attributes
    protected
    Definition Classes
    DeltaSource → DeltaSourceBase
  57. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  58. def logConsole(line: String): Unit
    Definition Classes
    DatabricksLogging
  59. def logDebug(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  60. def logDebug(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  61. def logError(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  62. def logError(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  63. def logInfo(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  64. def logInfo(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  65. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  66. def logTrace(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  67. def logTrace(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  68. def logWarning(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  69. def logWarning(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  70. val metadataPath: String
  71. val metadataTrackingLog: Option[DeltaSourceMetadataTrackingLog]
  72. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  73. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  74. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  75. val options: DeltaOptions
  76. val persistedMetadataAtSourceInit: Option[PersistedMetadata]

    The persisted schema from the schema log that must be used to read data files in this Delta streaming source.

    The persisted schema from the schema log that must be used to read data files in this Delta streaming source.

    Attributes
    protected
    Definition Classes
    DeltaSourceBase
  77. def prepareForTriggerAvailableNow(): Unit
    Definition Classes
    DeltaSourceBase → SupportsTriggerAvailableNow
  78. def productElementNames: Iterator[String]
    Definition Classes
    Product
  79. val readConfigurationsAtSourceInit: Map[String, String]
    Attributes
    protected
    Definition Classes
    DeltaSourceBase
  80. val readPartitionSchemaAtSourceInit: StructType
    Attributes
    protected
    Definition Classes
    DeltaSourceBase
  81. val readProtocolAtSourceInit: Protocol
    Attributes
    protected
    Definition Classes
    DeltaSourceBase
  82. val readSchemaAtSourceInit: StructType

    The read schema for this source during initialization, taking in account of SchemaLog.

    The read schema for this source during initialization, taking in account of SchemaLog.

    Attributes
    protected
    Definition Classes
    DeltaSourceBase
  83. lazy val readSnapshotDescriptor: SnapshotDescriptor

    Create a snapshot descriptor, customizing its metadata using metadata tracking if necessary

    Create a snapshot descriptor, customizing its metadata using metadata tracking if necessary

    Attributes
    protected
    Definition Classes
    DeltaSourceBase
  84. def readyToInitializeMetadataTrackingEagerly: Boolean

    Whether a schema tracking log is provided (and is empty), so we could initialize eagerly.

    Whether a schema tracking log is provided (and is empty), so we could initialize eagerly. This should only be used for the first write to the schema log, after then, schema tracking should not rely on this state any more.

    Attributes
    protected
    Definition Classes
    DeltaSourceMetadataEvolutionSupport
  85. def recordDeltaEvent(deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty, data: AnyRef = null, path: Option[Path] = None): Unit

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    path

    Used to log the path of the delta table when deltaLog is null.

    Attributes
    protected
    Definition Classes
    DeltaLogging
  86. def recordDeltaOperation[A](deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: => A): A

    Used to report the duration as well as the success or failure of an operation on a deltaLog.

    Used to report the duration as well as the success or failure of an operation on a deltaLog.

    Attributes
    protected
    Definition Classes
    DeltaLogging
  87. def recordDeltaOperationForTablePath[A](tablePath: String, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: => A): A

    Used to report the duration as well as the success or failure of an operation on a tahoePath.

    Used to report the duration as well as the success or failure of an operation on a tahoePath.

    Attributes
    protected
    Definition Classes
    DeltaLogging
  88. def recordEvent(metric: MetricDefinition, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit
    Definition Classes
    DatabricksLogging
  89. def recordFrameProfile[T](group: String, name: String)(thunk: => T): T
    Attributes
    protected
    Definition Classes
    DeltaLogging
  90. def recordOperation[S](opType: OpType, opTarget: String = null, extraTags: Map[TagDefinition, String], isSynchronous: Boolean = true, alwaysRecordStats: Boolean = false, allowAuthTags: Boolean = false, killJvmIfStuck: Boolean = false, outputMetric: MetricDefinition = METRIC_OPERATION_DURATION, silent: Boolean = true)(thunk: => S): S
    Definition Classes
    DatabricksLogging
  91. def recordProductEvent(metric: MetricDefinition with CentralizableMetric, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit
    Definition Classes
    DatabricksLogging
  92. def recordProductUsage(metric: MetricDefinition with CentralizableMetric, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit
    Definition Classes
    DatabricksLogging
  93. def recordUsage(metric: MetricDefinition, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit
    Definition Classes
    DatabricksLogging
  94. def reportLatestOffset(): Offset
    Definition Classes
    SupportsAdmissionControl
  95. val schema: StructType
    Definition Classes
    DeltaSourceBase → Source
  96. val snapshotAtSourceInit: SnapshotDescriptor
  97. val spark: SparkSession
  98. def stop(): Unit
    Definition Classes
    DeltaSource → SparkDataStream
  99. def stopIndexedFileIteratorAtSchemaChangeBarrier(fileActionScanIter: ClosableIterator[IndexedFile]): ClosableIterator[IndexedFile]

    This is called from getFileChangesWithRateLimit() during latestOffset().

    This is called from getFileChangesWithRateLimit() during latestOffset().

    Attributes
    protected
    Definition Classes
    DeltaSourceMetadataEvolutionSupport
  100. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  101. val tableId: String
    Attributes
    protected
  102. def toDeltaSourceOffset(offset: Offset): DeltaSourceOffset
  103. def toString(): String
    Definition Classes
    DeltaSource → AnyRef → Any
  104. def trackingMetadataChange: Boolean

    Whether this DeltaSource is utilizing a schema log entry as its read schema.

    Whether this DeltaSource is utilizing a schema log entry as its read schema.

    If user explicitly turn on the flag to fall back to using latest schema to read (i.e. the legacy mode), we will ignore the schema log.

    Attributes
    protected
    Definition Classes
    DeltaSourceMetadataEvolutionSupport
  105. def updateMetadataTrackingLogAndFailTheStreamIfNeeded(changedMetadataOpt: Option[Metadata], changedProtocolOpt: Option[Protocol], version: Long, replace: Boolean = false): Unit

    Write a new potentially changed metadata into the metadata tracking log.

    Write a new potentially changed metadata into the metadata tracking log. Then fail the stream to allow reanalysis if there are changes.

    changedMetadataOpt

    Potentially changed metadata action

    changedProtocolOpt

    Potentially changed protocol action

    version

    The version of change

    Attributes
    protected
    Definition Classes
    DeltaSourceMetadataEvolutionSupport
  106. def updateMetadataTrackingLogAndFailTheStreamIfNeeded(end: Offset): Unit

    Update the current stream schema in the schema tracking log and fail the stream.

    Update the current stream schema in the schema tracking log and fail the stream. This is called during commit(). It's ok to fail during commit() because in streaming's semantics, the batch with offset ending at end should've already being processed completely.

    Attributes
    protected
    Definition Classes
    DeltaSourceMetadataEvolutionSupport
  107. def validateCommitAndDecideSkipping(actions: Iterator[Action], version: Long, batchStartVersion: Long, batchEndVersionOpt: Option[Long] = None, verifyMetadataAction: Boolean = true): (Boolean, Option[Metadata], Option[Protocol])

    Check stream for violating any constraints.

    Check stream for violating any constraints.

    If verifyMetadataAction = true, we will break the stream when we detect any read-incompatible metadata changes.

    returns

    (true if commit should be skipped, a metadata action if found)

    Attributes
    protected
  108. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  109. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  110. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  111. def withStatusCode[T](statusCode: String, defaultMessage: String, data: Map[String, Any] = Map.empty)(body: => T): T

    Report a log to indicate some command is running.

    Report a log to indicate some command is running.

    Definition Classes
    DeltaProgressReporter
  112. object AdmissionLimits extends Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from DeltaSourceMetadataEvolutionSupport

Inherited from DeltaSourceCDCSupport

Inherited from DeltaSourceBase

Inherited from DeltaLogging

Inherited from DatabricksLogging

Inherited from DeltaProgressReporter

Inherited from Logging

Inherited from SupportsTriggerAvailableNow

Inherited from SupportsAdmissionControl

Inherited from Source

Inherited from SparkDataStream

Inherited from AnyRef

Inherited from Any

Ungrouped