Type Members

1.  trait AsyncLogPurge extends Logging

   Used to enable the capability to allow log purges to be done asynchronously

2.  class AsyncProgressTrackingMicroBatchExecution extends MicroBatchExecution

   Class to execute micro-batches when async progress tracking is enabled

3.  class AsyncStreamingQueryCheckpointMetadata extends StreamingQueryCheckpointMetadata

   A version of StreamingQueryCheckpointMetadata that supports async state checkpointing.

4.  class AvailableNowDataStreamWrapper extends SparkDataStream with SupportsTriggerAvailableNow with Logging

   This class wraps a SparkDataStream and makes it support Trigger.AvailableNow, by overriding its latestOffset method to always return the latest offset at the beginning of the query.

5.  class AvailableNowMicroBatchStreamWrapper extends AvailableNowDataStreamWrapper with MicroBatchStream

   This class wraps a MicroBatchStream and makes it supports Trigger.AvailableNow.

   This class wraps a MicroBatchStream and makes it supports Trigger.AvailableNow.

   See AvailableNowDataStreamWrapper for more details.

6.  class AvailableNowSourceWrapper extends AvailableNowDataStreamWrapper with Source

   This class wraps a Source and makes it supports Trigger.AvailableNow.

   This class wraps a Source and makes it supports Trigger.AvailableNow.

   See AvailableNowDataStreamWrapper for more details.

7.  abstract class CompactibleFileStreamLog[T <: AnyRef] extends checkpointing.HDFSMetadataLog[Array[T]]

   An abstract class for compactible metadata logs.

   An abstract class for compactible metadata logs. It will write one log file for each batch. The first line of the log file is the version number, and there are multiple serialized metadata lines following.

   As reading from many small files is usually pretty slow, also too many small files in one folder will mess the FS, CompactibleFileStreamLog will compact log files every 10 batches by default into a big file. When doing a compaction, it will read all old log files and merge them with the new batch.

8.  class ContinuousExecutionContext extends StreamExecutionContext

   Holds the all mutable state and metrics for a epoch when using continuous execution mode

9.  class ContinuousRecordEndpoint extends ThreadSafeRpcEndpoint

   A RPC end point for continuous readers to poll for records from the driver.

10.  case class ContinuousRecordPartitionOffset(partitionId: Int, offset: Int) extends PartitionOffset with Product with Serializable
11.  class ErrorNotifier extends Logging

    Class to notify of any errors that might have occurred out of band

12.  class FileStreamOptions extends Logging

    User specified options for file streams.

13.  class FileStreamSource extends SupportsAdmissionControl with SupportsTriggerAvailableNow with Source with Logging

    A very simple source that reads files from the given directory as they appear.

14.  class FileStreamSourceLog extends CompactibleFileStreamLog[FileEntry]
15.  case class FileStreamSourceOffset(logOffset: Long) extends Offset with Product with Serializable

    Offset for the FileStreamSource.

    Offset for the FileStreamSource.

    logOffset

    Position in the FileStreamSourceLog

16.  case class GetRecord(offset: ContinuousRecordPartitionOffset) extends Product with Serializable
17.  class IncrementalExecution extends QueryExecution with Logging

    A variant of QueryExecution that allows the execution of the given LogicalPlan plan incrementally.

    A variant of QueryExecution that allows the execution of the given LogicalPlan plan incrementally. Possibly preserving state in between each execution.

18.  case class LongOffset(offset: Long) extends Offset with Product with Serializable

    A simple offset for sources that produce a single linear stream of data.

19.  trait LowPriorityMemoryStreamImplicits extends AnyRef

    Provides lower-priority implicits for MemoryStream to prevent ambiguity when both SparkSession and SQLContext are in scope.

    Provides lower-priority implicits for MemoryStream to prevent ambiguity when both SparkSession and SQLContext are in scope. The implicits in the companion object, which use SparkSession, take higher precedence.

20.  case class MemoryStream[A](id: Int, sparkSession: SparkSession, numPartitions: Option[Int] = None)(implicit evidence$6: Encoder[A]) extends MemoryStreamBaseClass[A] with Product with Serializable

    A Source that produces value stored in memory as they are added by the user.

    A Source that produces value stored in memory as they are added by the user. This Source is intended for use in unit tests as it can only replay data when the object is still available.

    If numPartitions is provided, the rows will be redistributed to the given number of partitions.

21.  abstract class MemoryStreamBase[A] extends SparkDataStream

    A base class for memory stream implementations.

    A base class for memory stream implementations. Supports adding data and resetting.

22.  abstract class MemoryStreamBaseClass[A] extends MemoryStreamBase[A] with MicroBatchStream with SupportsTriggerAvailableNow with Logging
23.  class MemoryStreamInputPartition extends InputPartition
24.  class MemoryStreamScanBuilder extends ScanBuilder with Scan
25.  class MemoryStreamTable extends Table with SupportsRead
26.  class MetadataLogFileIndex extends PartitioningAwareFileIndex

    A FileIndex that generates the list of files to processing by reading them from the metadata log files generated by the FileStreamSink.

27.  class MetricsReporter extends metrics.source.Source with Logging

    Serves metrics from a org.apache.spark.sql.streaming.StreamingQuery to Codahale/DropWizard metrics

28.  class MicroBatchExecution extends StreamExecution with AsyncLogPurge
29.  class MicroBatchExecutionContext extends StreamExecutionContext with Logging

    Holds the all the mutable state and processing metrics for a single micro-batch when using micro batch execution mode.

30.  case class MicroBatchExecutionFailed() extends RuntimeException with Product with Serializable
31.  case class MicroBatchExecutionResult(isActive: Boolean, didExecute: Boolean) extends Product with Serializable
32.  case class MultiBatchExecutor() extends TriggerExecutor with Product with Serializable

    A trigger executor that runs multiple batches then terminates.

33.  sealed trait MultipleWatermarkPolicy extends AnyRef

    Policy to define how to choose a new global watermark value if there are multiple watermark operators in a streaming query.

34.  case class OffsetHolder(start: connector.read.streaming.Offset, end: Option[connector.read.streaming.Offset]) extends LogicalPlan with LeafNode with Product with Serializable
35.  case class ProcessingTimeExecutor(processingTimeTrigger: ProcessingTimeTrigger, clock: Clock = new SystemClock()) extends TriggerExecutor with Logging with Product with Serializable

    A trigger executor that runs a batch every intervalMs milliseconds.

36.  abstract class ProgressContext extends Logging

    This class holds variables and methods that are used track metrics and progress during the execution lifecycle of a batch that is being processed by the streaming query

37.  class ProgressReporter extends Logging

    Responsible for continually reporting statistics about the amount of data processed as well as latency for a streaming query.

    Responsible for continually reporting statistics about the amount of data processed as well as latency for a streaming query. This class is designed to hold information about a streaming query and contains methods that can be used on a streaming query, such as get the most recent progress of the query.

38.  class PropagateWatermarkSimulator extends WatermarkPropagator with Logging

    This implementation simulates propagation of watermark among operators.

    This implementation simulates propagation of watermark among operators.

    It is considered a "simulation" because watermarks are not being physically sent between operators, but rather propagated up the tree via post-order (children first) traversal of the query plan. This allows Structured Streaming to determine the new (input watermark, output watermark) for all nodes.

    For each node, below logic is applied:

    - Input watermark for specific node is decided by min(output watermarks from all children). -- Children providing no input watermark (DEFAULT_WATERMARK_MS) are excluded. -- If there is no valid input watermark from children, input watermark = DEFAULT_WATERMARK_MS. - Output watermark for specific node is decided as following: -- watermark nodes: origin watermark value This could be individual origin watermark value, but we decide to retain global watermark to keep the watermark model be simple. -- stateless nodes: same as input watermark -- stateful nodes: the return value of op.produceOutputWatermark(input watermark).

    See also

    StateStoreWriter.produceOutputWatermark Note that this implementation will throw an exception if watermark node sees a valid input watermark from children, meaning that we do not support re-definition of watermark. Once the algorithm traverses the physical plan tree, the association between stateful operator and input watermark will be constructed. Spark will request the input watermark for specific stateful operator, which this implementation will give the value from the association. We skip simulation of propagation for the value of watermark as 0. Input watermark for every operator will be 0. (This may not be expected for the case op.produceOutputWatermark returns higher than the input watermark, but it won't happen in most practical cases.)

39.  abstract class QueryExecutionThread extends UninterruptibleThread

    A special thread to run the stream query.

    A special thread to run the stream query. Some codes require to run in the QueryExecutionThread and will use classOf[QueryExecutionThread] to check.

40.  case class RateStreamOffset(partitionToValueAndRunTimeMs: Map[Int, ValueRunTimeMsPair]) extends connector.read.streaming.Offset with Product with Serializable
41.  case class SerializedOffset(json: String) extends Offset with Product with Serializable

    Used when loading a JSON serialized offset from external storage.

    Used when loading a JSON serialized offset from external storage. We are currently not responsible for converting JSON serialized data into an internal (i.e., object) representation. Sources should define a factory method in their source Offset companion objects that accepts a SerializedOffset for doing the conversion.

42.  case class SingleBatchExecutor() extends TriggerExecutor with Product with Serializable

    A trigger executor that runs a single batch only, then terminates.

43.  trait State extends AnyRef

    States for StreamExecution's lifecycle.

44.  abstract class StreamExecution extends StreamingQuery with Logging

    Manages the execution of a streaming Spark SQL query that is occurring in a separate thread.

    Manages the execution of a streaming Spark SQL query that is occurring in a separate thread. Unlike a standard query, a streaming query executes repeatedly each time new data arrives at any Source present in the query plan. Whenever new data arrives, a QueryExecution is created and the results are committed transactionally to the given Sink.

45.  abstract class StreamExecutionContext extends ProgressContext

    Holds the mutable state and metrics for a single batch for streaming query.

46.  case class StreamMetadata(id: String) extends Product with Serializable

    Contains metadata associated with a org.apache.spark.sql.streaming.StreamingQuery.

    Contains metadata associated with a org.apache.spark.sql.streaming.StreamingQuery. This information is written in the checkpoint location the first time a query is started and recovered every time the query is restarted.

    id

    unique id of the org.apache.spark.sql.streaming.StreamingQuery that needs to be persisted across restarts

47.  class StreamProgress extends Map[SparkDataStream, connector.read.streaming.Offset]

    A helper class that looks like a Map[Source, Offset].

48.  case class StreamingExecutionRelation(source: SparkDataStream, output: Seq[Attribute], catalogTable: Option[CatalogTable])(session: SparkSession) extends LogicalPlan with LeafNode with MultiInstanceRelation with Product with Serializable

    Used to link a streaming Source of data into a org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.

49.  class StreamingQueryCheckpointMetadata extends AnyRef

    An interface for accessing the checkpoint metadata associated with a streaming query.

50.  class StreamingQueryListenerBus extends SparkListener with ListenerBus[StreamingQueryListener, Event]

    A bus to forward events to StreamingQueryListeners.

    A bus to forward events to StreamingQueryListeners. This one will send received StreamingQueryListener.Events to the Spark listener bus. It also registers itself with Spark listener bus, so that it can receive StreamingQueryListener.Events and dispatch them to StreamingQueryListeners.

    Note 1: Each bus and its registered listeners are associated with a single SparkSession and StreamingQueryManager. So this bus will dispatch events to registered listeners for only those queries that were started in the associated SparkSession.

    Note 2: To rebuild Structured Streaming UI in SHS, this bus will be registered into org.apache.spark.scheduler.ReplayListenerBus. We check sparkListenerBus defined or not to determine how to process StreamingQueryListener.Event. If false, it means this bus is used to replay all streaming query event from eventLog.

51.  class StreamingQueryWrapper extends StreamingQuery with Serializable

    Wrap non-serializable StreamExecution to make the query serializable as it's easy for it to get captured with normal usage.

    Wrap non-serializable StreamExecution to make the query serializable as it's easy for it to get captured with normal usage. It's safe to capture the query but not use it in executors. However, if the user tries to call its methods, it will throw IllegalStateException.

52.  case class StreamingRelation(dataSource: DataSource, sourceName: String, output: Seq[Attribute]) extends LogicalPlan with LeafNode with MultiInstanceRelation with ExposesMetadataColumns with Product with Serializable

    Used to link a streaming DataSource into a org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.

    Used to link a streaming DataSource into a org.apache.spark.sql.catalyst.plans.logical.LogicalPlan. This is only used for creating a streaming org.apache.spark.sql.DataFrame from org.apache.spark.sql.DataFrameReader. It should be used to create Source and converted to StreamingExecutionRelation when passing to StreamExecution to run a query.

53.  case class StreamingRelationExec(sourceName: String, output: Seq[Attribute], tableIdentifier: Option[String]) extends SparkPlan with LeafExecNode with Product with Serializable

    A dummy physical plan for StreamingRelation to support org.apache.spark.sql.Dataset.explain

54.  trait TriggerExecutor extends AnyRef
55.  class UseSingleWatermarkPropagator extends WatermarkPropagator

    This implementation uses a single global watermark for late events and eviction.

    This implementation uses a single global watermark for late events and eviction.

    This implementation provides the behavior before Structured Streaming supports multiple stateful operators. (prior to SPARK-40925) This is only used for compatibility mode.

56.  case class ValueRunTimeMsPair(value: Long, runTimeMs: Long) extends Product with Serializable
57.  sealed trait WatermarkPropagator extends AnyRef

    Interface for propagating watermark.

    Interface for propagating watermark. The implementation is not required to be thread-safe, as all methods are expected to be called from the query execution thread. (The guarantee may change on further improvements on Structured Streaming - update implementations if we change the guarantee.)

58.  class WatermarkTracker extends Logging

    Tracks the watermark value of a streaming query based on a given policy