Type Members

1.  abstract class BaseStreamingDeduplicateExec extends SparkPlan with UnaryExecNode with StateStoreWriter with WatermarkSupport
2.  case class EventTimeStats(max: Long, min: Long, avg: Double, count: Long) extends Product with Serializable

   Class for collecting event time stats with an accumulator

3.  class EventTimeStatsAccum extends AccumulatorV2[Long, EventTimeStats]

   Accumulator that collects stats on event time in a batch.

4.  case class EventTimeWatermarkExec(nodeId: UUID, eventTime: Attribute, delay: CalendarInterval, child: SparkPlan) extends SparkPlan with UnaryExecNode with Product with Serializable

   Used to mark a column as the containing the event time for a given record.

   Used to mark a column as the containing the event time for a given record. In addition to adding appropriate metadata to this column, this operator also tracks the maximum observed event time. Based on the maximum observed time and a user specified delay, we can calculate the watermark after which we assume we will no longer see late records for a particular time period. Note that event time is measured in milliseconds.

5.  class MergingSortWithSessionWindowStateIterator extends Iterator[InternalRow] with Logging

   This class technically does the merge sort between input rows and existing sessions in state, to optimize the cost of sort on "input rows + existing sessions".

   This class technically does the merge sort between input rows and existing sessions in state, to optimize the cost of sort on "input rows + existing sessions". This is based on the precondition that input rows are sorted by "group keys + start time of session window".

   This only materializes the existing sessions into memory, which are tend to be not many per group key. The cost of sorting existing sessions would be also minor based on the assumption.

   The output rows are sorted with "group keys + start time of session window", which is same as the sort condition on input rows.

6.  trait SchemaValidationUtils extends Logging
7.  case class SessionWindowStateStoreRestoreExec(keyWithoutSessionExpressions: Seq[Attribute], sessionExpression: Attribute, stateInfo: Option[StatefulOperatorStateInfo], eventTimeWatermarkForLateEvents: Option[Long], eventTimeWatermarkForEviction: Option[Long], stateFormatVersion: Int, child: SparkPlan) extends SparkPlan with UnaryExecNode with StateStoreReader with WatermarkSupport with Product with Serializable

   This class sorts input rows and existing sessions in state and provides output rows as sorted by "group keys + start time of session window".

   This class sorts input rows and existing sessions in state and provides output rows as sorted by "group keys + start time of session window".

   Refer MergingSortWithSessionWindowStateIterator for more details.

8.  case class SessionWindowStateStoreSaveExec(keyWithoutSessionExpressions: Seq[Attribute], sessionExpression: Attribute, stateInfo: Option[StatefulOperatorStateInfo] = None, outputMode: Option[OutputMode] = None, eventTimeWatermarkForLateEvents: Option[Long] = None, eventTimeWatermarkForEviction: Option[Long] = None, stateFormatVersion: Int, child: SparkPlan) extends SparkPlan with UnaryExecNode with StateStoreWriter with WatermarkSupport with Product with Serializable

   This class replaces existing sessions for the grouping key with new sessions in state store.

   This class replaces existing sessions for the grouping key with new sessions in state store. All inputs are valid on storing into state store; don't filter out via watermark while storing. Refer the method doc of StreamingSessionWindowStateManager.updateSessions for more details.

   This class will provide the output according to the output mode. Update mode is not supported as the semantic is not feasible for session window.

9.  trait StateStoreReader extends SparkPlan with StatefulOperator with Logging

   An operator that reads from a StateStore.

10.  case class StateStoreRestoreExec(keyExpressions: Seq[Attribute], stateInfo: Option[StatefulOperatorStateInfo], stateFormatVersion: Int, child: SparkPlan) extends SparkPlan with UnaryExecNode with StateStoreReader with Product with Serializable

    For each input tuple, the key is calculated and the value from the StateStore is added to the stream (in addition to the input tuple) if present.

11.  case class StateStoreSaveExec(keyExpressions: Seq[Attribute], stateInfo: Option[StatefulOperatorStateInfo] = None, outputMode: Option[OutputMode] = None, eventTimeWatermarkForLateEvents: Option[Long] = None, eventTimeWatermarkForEviction: Option[Long] = None, stateFormatVersion: Int, child: SparkPlan) extends SparkPlan with UnaryExecNode with StateStoreWriter with WatermarkSupport with Product with Serializable

    For each input tuple, the key is calculated and the tuple is put into the StateStore.

12.  trait StateStoreWriter extends SparkPlan with StatefulOperator with PythonSQLMetrics with Logging

    An operator that writes to a StateStore.

13.  case class StatefulOpStateStoreCheckpointInfo(partitionId: Int, batchVersion: Long, stateStoreCkptId: Option[Array[String]], baseStateStoreCkptId: Option[Array[String]]) extends Product with Serializable

    Used to pass state store checkpoint information to load the correct state store checkpoint for a stateful operator.

    Used to pass state store checkpoint information to load the correct state store checkpoint for a stateful operator. Will be passed from the driver to exeuctors to load state store.

14.  trait StatefulOperator extends SparkPlan

    An operator that reads or writes state from the StateStore.

    An operator that reads or writes state from the StateStore. The StatefulOperatorStateInfo should be filled in by prepareForExecution in IncrementalExecution.

15.  trait StatefulOperatorCustomMetric extends AnyRef

    Custom stateful operator metric definition to allow operators to expose their own custom metrics.

    Custom stateful operator metric definition to allow operators to expose their own custom metrics. Also provides SQLMetric instance to show the metric in UI and accumulate it at the query level.

16.  case class StatefulOperatorCustomSumMetric(name: String, desc: String) extends StatefulOperatorCustomMetric with Product with Serializable

    Custom stateful operator metric for simple "count" gauge

17.  case class StatefulOperatorStateInfo(checkpointLocation: String, queryRunId: UUID, operatorId: Long, storeVersion: Long, numPartitions: Int, stateSchemaMetadata: Option[StateSchemaBroadcast] = None, stateStoreCkptIds: Option[Array[Array[String]]] = None) extends Product with Serializable

    Used to identify the state store for a given operator.

    Used to identify the state store for a given operator.

    stateStoreCkptIds is used to identify the checkpoint used for a specific stateful operator The basic workflow works as following: 1. When a stateful operator is created, it passes in the checkpoint IDs for each stateful operator through the StatefulOperatorStateInfo. 2. When a stateful task starts to execute, it will find the checkpointID for its shuffle partition and use it to recover the state store. The ID is eventually passed into the StateStore layer and eventually RocksDB State Store, where it is used to make sure the it loads the correct checkpoint 3. When the stateful task is finishing, after the state store is committed, the checkpoint ID is fetched from the state store by calling StateStore.getStateStoreCheckpointInfo() and added to the stateStoreCkptIds accumulator by calling StateStoreWriter.setStateStoreCheckpointInfo(). 4. When ending the batch, MicroBatchExecution calls each stateful operator's getStateStoreCheckpointInfo() which aggregates checkpointIDs from different partitions. The driver will persistent it into commit logs (not implemented yet). 5. When forming the next batch, the driver constructs the StatefulOperatorStateInfo with the checkpoint IDs for the previous batch.

18.  sealed trait StreamingAggregationStateManager extends Serializable

    Base trait for state manager purposed to be used from streaming aggregations.

19.  abstract class StreamingAggregationStateManagerBaseImpl extends StreamingAggregationStateManager
20.  class StreamingAggregationStateManagerImplV1 extends StreamingAggregationStateManagerBaseImpl

    The implementation of StreamingAggregationStateManager for state version 1.

    The implementation of StreamingAggregationStateManager for state version 1. In state version 1, the schema of key and value in state are follow:

    - key: Same as key expressions. - value: Same as input row attributes. The schema of value contains key expressions as well.

21.  class StreamingAggregationStateManagerImplV2 extends StreamingAggregationStateManagerBaseImpl

    The implementation of StreamingAggregationStateManager for state version 2.

    The implementation of StreamingAggregationStateManager for state version 2. In state version 2, the schema of key and value in state are follow:

    - key: Same as key expressions. - value: The diff between input row attributes and key expressions.

    The schema of value is changed to optimize the memory/space usage in state, via removing duplicated columns in key-value pair. Hence key columns are excluded from the schema of value.

22.  case class StreamingDeduplicateExec(keyExpressions: Seq[Attribute], child: SparkPlan, stateInfo: Option[StatefulOperatorStateInfo] = None, eventTimeWatermarkForLateEvents: Option[Long] = None, eventTimeWatermarkForEviction: Option[Long] = None) extends BaseStreamingDeduplicateExec with Product with Serializable

    Physical operator for executing streaming Deduplicate.

23.  case class StreamingDeduplicateWithinWatermarkExec(keyExpressions: Seq[Attribute], child: SparkPlan, stateInfo: Option[StatefulOperatorStateInfo] = None, eventTimeWatermarkForLateEvents: Option[Long] = None, eventTimeWatermarkForEviction: Option[Long] = None) extends BaseStreamingDeduplicateExec with Product with Serializable
24.  case class StreamingGlobalLimitExec(streamLimit: Long, child: SparkPlan, stateInfo: Option[StatefulOperatorStateInfo] = None, outputMode: Option[OutputMode] = None) extends SparkPlan with UnaryExecNode with StateStoreWriter with Product with Serializable

    A physical operator for executing a streaming limit, which makes sure no more than streamLimit rows are returned.

    A physical operator for executing a streaming limit, which makes sure no more than streamLimit rows are returned. This physical operator is only meant for logical limit operations that will get a input stream of rows that are effectively appends. For example, - limit on any query in append mode - limit before the aggregation in a streaming aggregation query complete mode

25.  case class StreamingLocalLimitExec(limit: Int, child: SparkPlan) extends SparkPlan with LimitExec with Product with Serializable

    A physical operator for executing limits locally on each partition.

    A physical operator for executing limits locally on each partition. The main difference from LocalLimitExec is that this will fully consume child plan's iterators to ensure that any stateful operation within child commits all the state changes (many stateful operations commit state changes only after the iterator is consumed).

26.  class StreamingSessionWindowHelper extends AnyRef
27.  sealed trait StreamingSessionWindowStateManager extends Serializable
28.  class StreamingSessionWindowStateManagerImplV1 extends StreamingSessionWindowStateManager with Logging
29.  case class UpdateEventTimeColumnExec(eventTime: Attribute, delay: CalendarInterval, eventTimeWatermarkForLateEvents: Option[Long], child: SparkPlan) extends SparkPlan with UnaryExecNode with Product with Serializable

    Updates the event time column to eventTime in the child output.

    Updates the event time column to eventTime in the child output. Any watermark calculations performed after this node will use the updated eventTimeColumn.

    This node also ensures that output emitted by the child node adheres to watermark. If the child node emits rows which are older than global watermark, the node will throw an query execution error and fail the user query.

30.  trait WatermarkSupport extends SparkPlan

    An operator that supports watermark.