Packages

  • package root
    Definition Classes
    root
  • package io
    Definition Classes
    root
  • package delta
    Definition Classes
    io
  • package sharing
    Definition Classes
    delta
  • package spark
    Definition Classes
    sharing
  • case class DeltaSharingSource(spark: SparkSession, deltaLog: RemoteDeltaLog, options: DeltaSharingOptions) extends Source with SupportsAdmissionControl with SupportsTriggerAvailableNow with Logging with Product with Serializable

    A streaming source for a Delta Sharing table.

    A streaming source for a Delta Sharing table.

    When a new stream is started, delta sharing starts by constructing a RemoteDeltaSnapshot at the current version of the table. This snapshot is broken up into batches until all existing data has been processed. Subsequent processing is done by tailing the change log looking for new data. This results in the streaming query returning the same answer as a batch query that had processed the entire dataset at any given point.

    Definition Classes
    spark
  • AdmissionLimits
  • LogStringContext

class AdmissionLimits extends AnyRef

Class that helps controlling how much data should be processed by a single micro-batch.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. AdmissionLimits
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new AdmissionLimits(maxFiles: Option[Int] = options.maxFilesPerTrigger, bytesToTake: Long = options.maxBytesPerTrigger.getOrElse(Long.MaxValue))

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def admit(fileAction: FileAction): Boolean

    Whether to admit the next file

  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. var bytesToTake: Long
  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  10. var filesToTake: Int
  11. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  12. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  13. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  14. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  15. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  16. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  17. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  18. def toReadLimit: ReadLimit
  19. def toString(): String
    Definition Classes
    AnyRef → Any
  20. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  21. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  22. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from AnyRef

Inherited from Any

Ungrouped