class MicroBatchExecutionContext extends StreamExecutionContext with Logging
Holds the all the mutable state and processing metrics for a single micro-batch when using micro batch execution mode.
- Alphabetic
- By Inheritance
- MicroBatchExecutionContext
- StreamExecutionContext
- ProgressContext
- Logging
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new MicroBatchExecutionContext(id: UUID, runId: UUID, name: String, triggerClock: Clock, sources: Seq[SparkDataStream], sink: Table, progressReporter: ProgressReporter, _batchId: Long, sparkSession: SparkSession, previousContext: Option[MicroBatchExecutionContext])
- _batchId
the id of this batch
- previousContext
the execution context of the previous micro-batch
Type Members
- implicit class LogStringContext extends AnyRef
- Definition Classes
- Logging
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- def MDC(key: LogKey, value: Any): MDC
- Attributes
- protected
- Definition Classes
- Logging
- var _batchId: Long
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- var batchId: Long
- Definition Classes
- StreamExecutionContext
- def carryOverExecStatsOnLatestExecutedBatch(): Unit
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- var currentStatus: StreamingQueryStatus
- Definition Classes
- ProgressContext
- var currentTriggerStartTimestamp: Long
- Attributes
- protected
- Definition Classes
- ProgressContext
- var endOffsets: StreamProgress
Stores the end offsets for this batch.
Stores the end offsets for this batch. Only the scheduler thread should modify this field, and only in atomic steps. Other threads should make a shallow copy if they are going to access this field more than once, since the field's value may change at any time.
- Definition Classes
- StreamExecutionContext
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- var execStatsOnLatestExecutedBatch: Option[ExecutionStats]
- Attributes
- protected
- Definition Classes
- ProgressContext
- var executionPlan: IncrementalExecution
- Definition Classes
- StreamExecutionContext
- def finishNoExecutionTrigger(lastExecutedEpochId: Long): Unit
Finalizes the trigger which did not execute a batch.
Finalizes the trigger which did not execute a batch.
- Definition Classes
- ProgressContext
- def finishTrigger(hasNewData: Boolean, lastExecution: IncrementalExecution, lastEpoch: Long): Unit
Override of finishTrigger to extract the map from IncrementalExecution.
Override of finishTrigger to extract the map from IncrementalExecution.
- Definition Classes
- ProgressContext
- def finishTrigger(hasNewData: Boolean, sourceToNumInputRowsMap: Map[SparkDataStream, Long], lastExecution: IncrementalExecution, lastEpochId: Long): Unit
Finalizes the query progress and adds it to list of recent status updates.
Finalizes the query progress and adds it to list of recent status updates.
- hasNewData
Whether the sources of this stream had new data for this trigger.
- Definition Classes
- ProgressContext
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def getDuration(key: String): Option[Long]
Retrieve a measured duration
Retrieve a measured duration
- Definition Classes
- ProgressContext
- def getNextContext(): MicroBatchExecutionContext
- def getStartTime(): Long
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- val id: UUID
- Definition Classes
- StreamExecutionContext
- def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- def initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
- var isCurrentBatchConstructed: Boolean
Signifies whether current batch (i.e.
Signifies whether current batch (i.e. for the batch
currentBatchId) has been constructed (i.e. written to the offsetLog) and is ready for execution. - final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- def isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- var lastTriggerStartTimestamp: Long
- Attributes
- protected
- Definition Classes
- ProgressContext
- var latestOffsets: StreamProgress
Tracks the latest offsets for each input source.
Tracks the latest offsets for each input source. Only the scheduler thread should modify this field, and only in atomic steps. Other threads should make a shallow copy if they are going to access this field more than once, since the field's value may change at any time.
- Definition Classes
- StreamExecutionContext
- def log: Logger
- Attributes
- protected
- Definition Classes
- Logging
- def logBasedOnLevel(level: Level)(f: => MessageWithContext): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logName: String
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- var metricWarningLogged: Boolean
Flag that signals whether any error with input metrics have already been logged
Flag that signals whether any error with input metrics have already been logged
- Attributes
- protected
- Definition Classes
- ProgressContext
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- var newData: Map[SparkDataStream, LogicalPlan]
Holds the most recent input data for each source.
Holds the most recent input data for each source.
- Definition Classes
- StreamExecutionContext → ProgressContext
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- var offsetSeqMetadata: OffsetSeqMetadata
Metadata associated with the offset seq of a batch in the query.
Metadata associated with the offset seq of a batch in the query.
- Definition Classes
- StreamExecutionContext → ProgressContext
- def onExecutionComplete(): Unit
- Definition Classes
- MicroBatchExecutionContext → StreamExecutionContext
- def onExecutionFailure(): Unit
- Definition Classes
- MicroBatchExecutionContext → StreamExecutionContext
- var previousContext: Option[MicroBatchExecutionContext]
- def recordEndOffsets(to: StreamProgress): Unit
Only used by Real-time Mode.
Only used by Real-time Mode. For other cases, end offsets are determined in the batch planning phase so it is never need to be updated.
- Definition Classes
- ProgressContext
- def recordTriggerOffsets(from: StreamProgress, to: StreamProgress, latest: StreamProgress): Unit
Record the offsets range this trigger will process.
Record the offsets range this trigger will process. Call this before updating
committedOffsetsinStreamExecutionto make sure that the correct range is recorded.- Definition Classes
- ProgressContext
- def reportTimeTaken(triggerDetailKey: String, timeTakenMs: Long): Unit
Reports an input duration for a particular detail key in the next query progress update.
Reports an input duration for a particular detail key in the next query progress update. Can be used directly instead of reportTimeTaken(key)(body) when the duration is measured asynchronously.
- Definition Classes
- ProgressContext
- def reportTimeTaken[T](triggerDetailKey: String)(body: => T): T
Records the duration of running
bodyfor the next query progress update.Records the duration of running
bodyfor the next query progress update.- Definition Classes
- ProgressContext
- var sinkCommitProgress: Option[StreamWriterCommitProgress]
- Definition Classes
- ProgressContext
- var startOffsets: StreamProgress
Stores the start offset for this batch.
Stores the start offset for this batch. Only the scheduler thread should modify this field, and only in atomic steps. Other threads should make a shallow copy if they are going to access this field more than once, since the field's value may change at any time.
- Definition Classes
- StreamExecutionContext
- def startTrigger(): Unit
Begins recording statistics about query progress for a given trigger.
Begins recording statistics about query progress for a given trigger.
- Definition Classes
- MicroBatchExecutionContext → ProgressContext
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- MicroBatchExecutionContext → AnyRef → Any
- def updateStatusMessage(message: String): Unit
Updates the message returned in
status.Updates the message returned in
status.- Definition Classes
- ProgressContext
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- def withLogContext(context: Map[String, String])(body: => Unit): Unit
- Attributes
- protected
- Definition Classes
- Logging
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)