org.apache.spark.sql.execution.streaming.continuous
ContinuousExecution
Companion object ContinuousExecution
class ContinuousExecution extends StreamExecution
- Alphabetic
- By Inheritance
- ContinuousExecution
- StreamExecution
- Logging
- StreamingQuery
- StreamingQuery
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new ContinuousExecution(sparkSession: classic.SparkSession, trigger: Trigger, triggerClock: Clock, extraOptions: Map[String, String], plan: WriteToStream)
Type Members
- implicit class LogStringContext extends AnyRef
- Definition Classes
- Logging
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- def MDC(key: LogKey, value: Any): MDC
- Attributes
- protected
- Definition Classes
- Logging
- def addOffset(epoch: Long, stream: ContinuousStream, partitionOffsets: Seq[PartitionOffset]): Unit
Report ending partition offsets for the given reader at the given epoch.
- val analyzedPlan: LogicalPlan
- Definition Classes
- StreamExecution
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def availableOffsets: StreamProgress
Get the end or formerly know as "available" offsets of the latest batch that has been planned
Get the end or formerly know as "available" offsets of the latest batch that has been planned
- Definition Classes
- StreamExecution
- def awaitInitialization(timeoutMs: Long): Unit
Await until all fields of the query have been initialized.
Await until all fields of the query have been initialized.
- Definition Classes
- StreamExecution
- val awaitProgressLock: ReentrantLock
A lock used to wait/notify when batches complete.
A lock used to wait/notify when batches complete. Use a fair lock to avoid thread starvation.
- Attributes
- protected
- Definition Classes
- StreamExecution
- val awaitProgressLockCondition: Condition
- Attributes
- protected
- Definition Classes
- StreamExecution
- def awaitTermination(timeoutMs: Long): Boolean
- Definition Classes
- StreamExecution → StreamingQuery
- def awaitTermination(): Unit
- Definition Classes
- StreamExecution → StreamingQuery
- def checkpointFile(name: String): String
Returns the path of a file with
namein the checkpoint directory.Returns the path of a file with
namein the checkpoint directory.- Attributes
- protected
- Definition Classes
- StreamExecution
- val checkpointMetadata: StreamingQueryCheckpointMetadata
Manages the metadata from this checkpoint location.
Manages the metadata from this checkpoint location.
- Attributes
- protected
- Definition Classes
- StreamExecution
- def cleanup(): Unit
Any clean up that needs to happen when the query is stopped or exits
Any clean up that needs to happen when the query is stopped or exits
- Attributes
- protected
- Definition Classes
- StreamExecution
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- def commit(epoch: Long): Unit
Mark the specified epoch as committed.
Mark the specified epoch as committed. All readers must have reported end offsets for the epoch before this is called.
- lazy val commitLog: CommitLog
- Definition Classes
- StreamExecution
- var committedOffsets: StreamProgress
Tracks how much data we have processed and committed to the sink or state store from each input source.
Tracks how much data we have processed and committed to the sink or state store from each input source. Only the scheduler thread should modify this field, and only in atomic steps. Other threads should make a shallow copy if they are going to access this field more than once, since the field's value may change at any time.
- Definition Classes
- StreamExecution
- def createWrite(table: SupportsWrite, options: Map[String, String], inputPlan: LogicalPlan): Write
- Attributes
- protected
- Definition Classes
- StreamExecution
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def exception: Option[StreamingQueryException]
Returns the StreamingQueryException if the query was terminated by an exception.
Returns the StreamingQueryException if the query was terminated by an exception.
- Definition Classes
- StreamExecution → StreamingQuery
- def explain(): Unit
- Definition Classes
- StreamExecution → StreamingQuery
- def explain(extended: Boolean): Unit
- Definition Classes
- StreamExecution → StreamingQuery
- def explainInternal(extended: Boolean): String
Expose for tests
Expose for tests
- Definition Classes
- StreamExecution
- def getBatchDescriptionString: String
- Attributes
- protected
- Definition Classes
- StreamExecution
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def getLatestExecutionContext(): StreamExecutionContext
Get the latest execution context .
Get the latest execution context .
- Definition Classes
- ContinuousExecution → StreamExecution
- def getStartOffsetsOfLatestBatch: StreamProgress
Get the start offsets of the latest batch that has been planned
Get the start offsets of the latest batch that has been planned
- Definition Classes
- StreamExecution
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- val id: UUID
- Definition Classes
- StreamExecution → StreamingQuery
- def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- def initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def interruptAndAwaitExecutionThreadTermination(): Unit
Interrupts the query execution thread and awaits its termination until until it exceeds the timeout.
Interrupts the query execution thread and awaits its termination until until it exceeds the timeout. The timeout can be set on "spark.sql.streaming.stopTimeout".
- Attributes
- protected
- Definition Classes
- StreamExecution
- Annotations
- @throws("")
- Exceptions thrown
TimeoutExceptionIf the thread cannot be stopped within the timeout
- def isActive: Boolean
Whether the query is currently active or not
Whether the query is currently active or not
- Definition Classes
- StreamExecution → StreamingQuery
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- def isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- def lastExecution: IncrementalExecution
- Definition Classes
- StreamExecution
- def lastProgress: StreamingQueryProgress
- Definition Classes
- StreamExecution → StreamingQuery
- def latestOffsets: StreamProgress
- Definition Classes
- StreamExecution
- def log: Logger
- Attributes
- protected
- Definition Classes
- Logging
- def logBasedOnLevel(level: Level)(f: => MessageWithContext): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logName: String
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- var loggingThreadContext: Instance
- Attributes
- protected
- Definition Classes
- StreamExecution
- val logicalPlan: WriteToContinuousDataSource
The base logical plan which will be used across batch runs.
The base logical plan which will be used across batch runs. Once the value is set, it should not be modified.
- Definition Classes
- ContinuousExecution → StreamExecution
- val minLogEntriesToMaintain: Int
- Attributes
- protected
- Definition Classes
- StreamExecution
- val name: String
- Definition Classes
- StreamExecution → StreamingQuery
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- var noNewData: Boolean
A flag to indicate that a batch has completed with no new data available.
A flag to indicate that a batch has completed with no new data available.
- Attributes
- protected
- Definition Classes
- StreamExecution
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- lazy val offsetLog: OffsetSeqLog
- Definition Classes
- StreamExecution
- val outputMode: OutputMode
- Definition Classes
- StreamExecution
- val pollingDelayMs: Long
- Attributes
- protected
- Definition Classes
- StreamExecution
- def postEvent(event: Event): Unit
- Attributes
- protected
- Definition Classes
- StreamExecution
- val prettyIdString: String
Pretty identified string of printing in logs.
Pretty identified string of printing in logs. Format is If name is set "queryName [id = xyz, runId = abc]" else "[id = xyz, runId = abc]"
- Attributes
- protected
- Definition Classes
- StreamExecution
- def processAllAvailable(): Unit
- Definition Classes
- StreamExecution → StreamingQuery
- val progressReporter: ProgressReporter
- Attributes
- protected
- Definition Classes
- StreamExecution
- def purge(threshold: Long): Unit
- Attributes
- protected
- Definition Classes
- StreamExecution
- def purgeStatefulMetadata(plan: SparkPlan): Unit
- Attributes
- protected
- Definition Classes
- StreamExecution
- val queryExecutionThread: QueryExecutionThread
The thread that runs the micro-batches of this stream.
The thread that runs the micro-batches of this stream. Note that this thread must be org.apache.spark.util.UninterruptibleThread to workaround KAFKA-1894: interrupting a running
KafkaConsumermay cause endless loop.- Definition Classes
- StreamExecution
- def recentProgress: Array[StreamingQueryProgress]
- Definition Classes
- StreamExecution → StreamingQuery
- val resolvedCheckpointRoot: String
- Definition Classes
- StreamExecution
- def runActivatedStream(sparkSessionForStream: classic.SparkSession): Unit
Run the activated stream until stopped.
Run the activated stream until stopped.
- Attributes
- protected
- Definition Classes
- ContinuousExecution → StreamExecution
- val runId: UUID
- Definition Classes
- StreamExecution → StreamingQuery
- val sink: Table
- Definition Classes
- StreamExecution
- var sources: Seq[ContinuousStream]
The list of stream instances which will be used across batch runs.
The list of stream instances which will be used across batch runs. Once the value is set, it should not be modified.
- Attributes
- protected
- Definition Classes
- ContinuousExecution → StreamExecution
- val sparkSession: classic.SparkSession
<invalid inheritdoc annotation>
<invalid inheritdoc annotation>
- Definition Classes
- StreamExecution → StreamingQuery → StreamingQuery
- val sparkSessionForStream: classic.SparkSession
Isolated spark session to run the batches with.
Isolated spark session to run the batches with.
- Attributes
- protected[sql]
- Definition Classes
- StreamExecution
- def start(): Unit
Starts the execution.
Starts the execution. This returns only after the thread has started and QueryStartedEvent has been posted to all the listeners.
- Definition Classes
- StreamExecution
- val state: AtomicReference[State]
Defines the internal state of execution
Defines the internal state of execution
- Attributes
- protected
- Definition Classes
- StreamExecution
- def status: StreamingQueryStatus
- Definition Classes
- StreamExecution → StreamingQuery
- def stop(): Unit
Stops the query execution thread to terminate the query.
Stops the query execution thread to terminate the query.
- Definition Classes
- ContinuousExecution → StreamingQuery
- def stopInNewThread(error: Throwable): Unit
Stores error and stops the query execution thread to terminate the query in new thread.
- def stopSources(): Unit
Stops all streaming sources safely.
Stops all streaming sources safely.
- Attributes
- protected
- Definition Classes
- StreamExecution
- var streamDeathCause: StreamingQueryException
- Attributes
- protected
- Definition Classes
- StreamExecution
- lazy val streamMetrics: MetricsReporter
Used to report metrics to coda-hale.
Used to report metrics to coda-hale. This uses id for easier tracking across restarts.
- Definition Classes
- StreamExecution
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- StreamExecution → AnyRef → Any
- val trigger: Trigger
- Definition Classes
- StreamExecution
- val triggerClock: Clock
- Definition Classes
- StreamExecution
- var uniqueSources: Map[SparkDataStream, ReadLimit]
A list of unique sources in the query plan.
A list of unique sources in the query plan. This will be set when generating logical plan.
- Attributes
- protected
- Definition Classes
- StreamExecution
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- val watermarkMsMap: Map[Int, Long]
A map of current watermarks, keyed by the position of the watermark operator in the physical plan.
A map of current watermarks, keyed by the position of the watermark operator in the physical plan.
This state is 'soft state', which does not affect the correctness and semantics of watermarks and is not persisted across query restarts. The fault-tolerant watermark state is in offsetSeqMetadata.
- Attributes
- protected
- Definition Classes
- StreamExecution
- def withLogContext(context: Map[String, String])(body: => Unit): Unit
- Attributes
- protected
- Definition Classes
- Logging
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)