org.apache.spark.sql.execution.streaming
StatefulOperatorStateInfo
Companion object StatefulOperatorStateInfo
case class StatefulOperatorStateInfo(checkpointLocation: String, queryRunId: UUID, operatorId: Long, storeVersion: Long, numPartitions: Int, stateSchemaMetadata: Option[StateSchemaBroadcast] = None, stateStoreCkptIds: Option[Array[Array[String]]] = None) extends Product with Serializable
Used to identify the state store for a given operator.
stateStoreCkptIds is used to identify the checkpoint used for a specific stateful operator The basic workflow works as following: 1. When a stateful operator is created, it passes in the checkpoint IDs for each stateful operator through the StatefulOperatorStateInfo. 2. When a stateful task starts to execute, it will find the checkpointID for its shuffle partition and use it to recover the state store. The ID is eventually passed into the StateStore layer and eventually RocksDB State Store, where it is used to make sure the it loads the correct checkpoint 3. When the stateful task is finishing, after the state store is committed, the checkpoint ID is fetched from the state store by calling StateStore.getStateStoreCheckpointInfo() and added to the stateStoreCkptIds accumulator by calling StateStoreWriter.setStateStoreCheckpointInfo(). 4. When ending the batch, MicroBatchExecution calls each stateful operator's getStateStoreCheckpointInfo() which aggregates checkpointIDs from different partitions. The driver will persistent it into commit logs (not implemented yet). 5. When forming the next batch, the driver constructs the StatefulOperatorStateInfo with the checkpoint IDs for the previous batch.
- Alphabetic
- By Inheritance
- StatefulOperatorStateInfo
- Serializable
- Product
- Equals
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new StatefulOperatorStateInfo(checkpointLocation: String, queryRunId: UUID, operatorId: Long, storeVersion: Long, numPartitions: Int, stateSchemaMetadata: Option[StateSchemaBroadcast] = None, stateStoreCkptIds: Option[Array[Array[String]]] = None)
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- val checkpointLocation: String
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def getStateStoreCkptId(partitionId: Int): Option[Array[String]]
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- val numPartitions: Int
- val operatorId: Long
- def productElementNames: Iterator[String]
- Definition Classes
- Product
- val queryRunId: UUID
- val stateSchemaMetadata: Option[StateSchemaBroadcast]
- val stateStoreCkptIds: Option[Array[Array[String]]]
- val storeVersion: Long
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- StatefulOperatorStateInfo → AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)