Packages

case class StatefulOperatorStateInfo(checkpointLocation: String, queryRunId: UUID, operatorId: Long, storeVersion: Long, numPartitions: Int, stateSchemaMetadata: Option[StateSchemaBroadcast] = None, stateStoreCkptIds: Option[Array[Array[String]]] = None) extends Product with Serializable

Used to identify the state store for a given operator.

stateStoreCkptIds is used to identify the checkpoint used for a specific stateful operator The basic workflow works as following: 1. When a stateful operator is created, it passes in the checkpoint IDs for each stateful operator through the StatefulOperatorStateInfo. 2. When a stateful task starts to execute, it will find the checkpointID for its shuffle partition and use it to recover the state store. The ID is eventually passed into the StateStore layer and eventually RocksDB State Store, where it is used to make sure the it loads the correct checkpoint 3. When the stateful task is finishing, after the state store is committed, the checkpoint ID is fetched from the state store by calling StateStore.getStateStoreCheckpointInfo() and added to the stateStoreCkptIds accumulator by calling StateStoreWriter.setStateStoreCheckpointInfo(). 4. When ending the batch, MicroBatchExecution calls each stateful operator's getStateStoreCheckpointInfo() which aggregates checkpointIDs from different partitions. The driver will persistent it into commit logs (not implemented yet). 5. When forming the next batch, the driver constructs the StatefulOperatorStateInfo with the checkpoint IDs for the previous batch.

Linear Supertypes
Serializable, Product, Equals, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. StatefulOperatorStateInfo
  2. Serializable
  3. Product
  4. Equals
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new StatefulOperatorStateInfo(checkpointLocation: String, queryRunId: UUID, operatorId: Long, storeVersion: Long, numPartitions: Int, stateSchemaMetadata: Option[StateSchemaBroadcast] = None, stateStoreCkptIds: Option[Array[Array[String]]] = None)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. val checkpointLocation: String
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  9. def getStateStoreCkptId(partitionId: Int): Option[Array[String]]
  10. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  11. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  12. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  13. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  14. val numPartitions: Int
  15. val operatorId: Long
  16. def productElementNames: Iterator[String]
    Definition Classes
    Product
  17. val queryRunId: UUID
  18. val stateSchemaMetadata: Option[StateSchemaBroadcast]
  19. val stateStoreCkptIds: Option[Array[Array[String]]]
  20. val storeVersion: Long
  21. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  22. def toString(): String
    Definition Classes
    StatefulOperatorStateInfo → AnyRef → Any
  23. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  24. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  25. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from AnyRef

Inherited from Any

Ungrouped