Packages

class IncrementalExecution extends QueryExecution with Logging

A variant of QueryExecution that allows the execution of the given LogicalPlan plan incrementally. Possibly preserving state in between each execution.

Linear Supertypes
QueryExecution, Logging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. IncrementalExecution
  2. QueryExecution
  3. Logging
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new IncrementalExecution(sparkSession: classic.SparkSession, logicalPlan: LogicalPlan, outputMode: OutputMode, checkpointLocation: String, queryId: UUID, runId: UUID, currentBatchId: Long, prevOffsetSeqMetadata: Option[OffsetSeqMetadata], offsetSeqMetadata: OffsetSeqMetadata, watermarkPropagator: WatermarkPropagator, isFirstBatch: Boolean, currentStateStoreCkptId: Map[Long, Array[Array[String]]] = MutableMap[Long, Array[Array[String]]](), stateSchemaMetadatas: Map[Long, StateSchemaBroadcast] = MutableMap[Long, StateSchemaBroadcast](), mode: CommandExecutionMode.Value = CommandExecutionMode.ALL, isTerminatingTrigger: Boolean = false)

    currentStateStoreCkptId

    checkpoint ID for the latest committed version. It is operatorID -> array of checkpointIDs. Array index n represents checkpoint ID for the nth shuffle partition.

    stateSchemaMetadatas

    map that keeps track of the StateSchemaBroadcasts per operator. It is passed in by reference from MicrobatchExecution, so it is populated during planning in the first batch, and passed in to the StatefulOperatorStateInfo on every subsequent batch

Type Members

  1. implicit class LogStringContext extends AnyRef
    Definition Classes
    Logging
  2. sealed trait SparkPlanPartialRule extends AnyRef

Value Members

  1. object debug

    A special namespace for commands that can be used to debug query execution.

    A special namespace for commands that can be used to debug query execution.

    Definition Classes
    QueryExecution
  2. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  3. final def ##: Int
    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  5. def MDC(key: LogKey, value: Any): MDC
    Attributes
    protected
    Definition Classes
    Logging
  6. def analyzed: LogicalPlan
    Definition Classes
    QueryExecution
  7. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  8. def assertAnalyzed(): Unit

    no need to try-catch again as this is already done once

    no need to try-catch again as this is already done once

    Definition Classes
    IncrementalExecution → QueryExecution
  9. def assertCommandExecuted(): Unit
    Definition Classes
    QueryExecution
  10. def assertExecutedPlanPrepared(): Unit
    Definition Classes
    QueryExecution
  11. def assertOptimized(): Unit
    Definition Classes
    QueryExecution
  12. def assertSparkPlanPrepared(): Unit
    Definition Classes
    QueryExecution
  13. def assertSupported(): Unit

    No need assert supported, as this check has already been done

    No need assert supported, as this check has already been done

    Definition Classes
    IncrementalExecution → QueryExecution
  14. val checkpointLocation: String
  15. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  16. def commandExecuted: LogicalPlan
    Definition Classes
    QueryExecution
  17. val currentBatchId: Long
  18. val currentStateStoreCkptId: Map[Long, Array[Array[String]]]
  19. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  20. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  21. def executePhase[T](phase: String)(block: => T): T
    Attributes
    protected
    Definition Classes
    QueryExecution
  22. def executedPlan: SparkPlan
    Definition Classes
    QueryExecution
  23. def explainString(mode: ExplainMode): String
    Definition Classes
    QueryExecution
  24. def extendedExplainInfo(append: (String) => Unit, plan: SparkPlan): Unit
    Definition Classes
    QueryExecution
  25. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  26. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  27. val id: Long
    Definition Classes
    QueryExecution
  28. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  29. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  30. val isFirstBatch: Boolean
  31. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  32. lazy val isLazyAnalysis: Boolean
    Definition Classes
    QueryExecution
  33. def isSqlScript: Boolean

    Check whether the query represented by this QueryExecution is a SQL script.

    Check whether the query represented by this QueryExecution is a SQL script.

    returns

    True if the query is a SQL script, False otherwise.

    Definition Classes
    QueryExecution
  34. val isTerminatingTrigger: Boolean
  35. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  36. val lazyToRdd: LazyTry[SQLExecutionRDD]
    Definition Classes
    QueryExecution
  37. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  38. def logBasedOnLevel(level: Level)(f: => MessageWithContext): Unit
    Attributes
    protected
    Definition Classes
    Logging
  39. def logDebug(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  40. def logDebug(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  41. def logDebug(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  42. def logDebug(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  43. def logError(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  44. def logError(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  45. def logError(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  46. def logError(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  47. def logInfo(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  48. def logInfo(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  49. def logInfo(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  50. def logInfo(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  51. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  52. def logTrace(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  53. def logTrace(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  54. def logTrace(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  55. def logTrace(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  56. def logWarning(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  57. def logWarning(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  58. def logWarning(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  59. def logWarning(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  60. val logical: LogicalPlan
    Definition Classes
    QueryExecution
  61. val mode: CommandExecutionMode.Value
    Definition Classes
    QueryExecution
  62. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  63. def normalized: LogicalPlan
    Definition Classes
    QueryExecution
  64. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  65. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  66. def observedMetrics: Map[String, Row]

    Get the metrics observed during the execution of the query plan.

    Get the metrics observed during the execution of the query plan.

    Definition Classes
    QueryExecution
    Annotations
    @GuardedBy()
  67. val offsetSeqMetadata: OffsetSeqMetadata
  68. lazy val optimizedPlan: LogicalPlan

    See [SPARK-18339] Walk the optimized logical plan and replace CurrentBatchTimestamp with the desired literal

    See [SPARK-18339] Walk the optimized logical plan and replace CurrentBatchTimestamp with the desired literal

    Definition Classes
    IncrementalExecution → QueryExecution
  69. val outputMode: OutputMode
  70. val planner: SparkPlanner
    Definition Classes
    IncrementalExecution → QueryExecution
  71. def preparations: Seq[Rule[SparkPlan]]
    Definition Classes
    IncrementalExecution → QueryExecution
  72. val prevOffsetSeqMetadata: Option[OffsetSeqMetadata]
  73. val queryId: UUID
  74. val refreshPhaseEnabled: Boolean
    Definition Classes
    QueryExecution
  75. val runId: UUID
  76. def shouldRunAnotherBatch(newMetadata: OffsetSeqMetadata): Boolean

    Should the MicroBatchExecution run another batch based on this execution and the current updated metadata.

    Should the MicroBatchExecution run another batch based on this execution and the current updated metadata.

    This method performs simulation of watermark propagation against new batch (which is not planned yet), which is required for asking the needs of another batch to each stateful operator.

  77. val shuffleCleanupMode: ShuffleCleanupMode
    Definition Classes
    QueryExecution
  78. def simpleString: String
    Definition Classes
    QueryExecution
  79. def sparkPlan: SparkPlan
    Definition Classes
    QueryExecution
  80. val sparkSession: classic.SparkSession
    Definition Classes
    QueryExecution
  81. val state: Rule[SparkPlan]
  82. val stateSchemaMetadatas: Map[Long, StateSchemaBroadcast]
  83. def stringWithStats: String
    Definition Classes
    QueryExecution
  84. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  85. def toRdd: RDD[InternalRow]

    Internal version of the RDD.

    Internal version of the RDD. Avoids copies and has no schema. Note for callers: Spark may apply various optimization including reusing object: this means the row is valid only for the iteration it is retrieved. You should avoid storing row and accessing after iteration. (Calling collect() is one of known bad usage.) If you want to store these rows into collection, please apply some converter or copy row which produces new object per iteration. Given QueryExecution is not a public class, end users are discouraged to use this: please use Dataset.rdd instead where conversion will be applied.

    Definition Classes
    QueryExecution
  86. def toString(): String
    Definition Classes
    QueryExecution → AnyRef → Any
  87. val tracker: QueryPlanningTracker
    Definition Classes
    QueryExecution
  88. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  89. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  90. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  91. val watermarkPropagator: WatermarkPropagator
  92. def withCachedData: LogicalPlan
    Definition Classes
    QueryExecution
  93. def withLogContext(context: Map[String, String])(body: => Unit): Unit
    Attributes
    protected
    Definition Classes
    Logging
  94. object ConvertLocalLimitRule extends SparkPlanPartialRule
  95. object ShufflePartitionsRule extends SparkPlanPartialRule
  96. object StateOpIdRule extends SparkPlanPartialRule
  97. object StateSchemaAndOperatorMetadataRule extends SparkPlanPartialRule
  98. object WatermarkPropagationRule extends SparkPlanPartialRule

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from QueryExecution

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped