case class DeltaSharingSource(spark: SparkSession, deltaLog: RemoteDeltaLog, options: DeltaSharingOptions) extends Source with SupportsAdmissionControl with SupportsTriggerAvailableNow with Logging with Product with Serializable
A streaming source for a Delta Sharing table.
When a new stream is started, delta sharing starts by constructing a RemoteDeltaSnapshot at the current version of the table. This snapshot is broken up into batches until all existing data has been processed. Subsequent processing is done by tailing the change log looking for new data. This results in the streaming query returning the same answer as a batch query that had processed the entire dataset at any given point.
Linear Supertypes
Ordering
- Alphabetic
- By Inheritance
Inherited
- DeltaSharingSource
- Serializable
- Product
- Equals
- Logging
- SupportsTriggerAvailableNow
- SupportsAdmissionControl
- Source
- SparkDataStream
- AnyRef
- Any
- Hide All
- Show All
Visibility
- Public
- Protected
Instance Constructors
- new DeltaSharingSource(spark: SparkSession, deltaLog: RemoteDeltaLog, options: DeltaSharingOptions)
Type Members
- class AdmissionLimits extends AnyRef
Class that helps controlling how much data should be processed by a single micro-batch.
- implicit class LogStringContext extends AnyRef
- Definition Classes
- Logging
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- def commit(end: Offset): Unit
- Definition Classes
- Source → SparkDataStream
- def commit(end: Offset): Unit
- Definition Classes
- Source
- val deltaLog: RemoteDeltaLog
- def deserializeOffset(json: String): Offset
- Definition Classes
- Source → SparkDataStream
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def getBatch(startOffsetOption: Option[Offset], end: Offset): DataFrame
- Definition Classes
- DeltaSharingSource → Source
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def getDefaultReadLimit(): ReadLimit
- Definition Classes
- DeltaSharingSource → SupportsAdmissionControl
- def getOffset: Option[Offset]
- Definition Classes
- DeltaSharingSource → Source
- def initialOffset(): Offset
- Definition Classes
- Source → SparkDataStream
- def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- def initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- def isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- def latestOffset(startOffset: Offset, limit: ReadLimit): Offset
- Definition Classes
- DeltaSharingSource → SupportsAdmissionControl
- def log: Logger
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logName: String
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- val options: DeltaSharingOptions
- def prepareForTriggerAvailableNow(): Unit
Capture the server version once at query start.
Capture the server version once at query start. All subsequent calls to getOrUpdateLatestTableVersion return this frozen value, which in turn caps endingVersionForQuery in maybeGetFileChanges. Once all files up to the frozen version have been consumed, getBatch returns no new data and the AvailableNow engine terminates the query.
No additional guard inside maybeGetFileChanges is needed (unlike DeltaFormatSharingSource's needNewFilesFromServer): that source has an intermediate local delta log which must be explicitly stopped from growing past the frozen version; here there is no such buffer -- sortedFetchedFiles is populated directly from the server RPC whose version ceiling is already frozen.
- Definition Classes
- DeltaSharingSource → SupportsTriggerAvailableNow
- def productElementNames: Iterator[String]
- Definition Classes
- Product
- def reportLatestOffset(): Offset
- Definition Classes
- SupportsAdmissionControl
- val schema: StructType
- Definition Classes
- DeltaSharingSource → Source
- val spark: SparkSession
- def stop(): Unit
- Definition Classes
- DeltaSharingSource → SparkDataStream
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- DeltaSharingSource → AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- def withLogContext(context: Map[String, String])(body: => Unit): Unit
- Attributes
- protected
- Definition Classes
- Logging
- object AdmissionLimits
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)