org.apache.spark.sql.execution.streaming.state
StateStoreProvider
Companion object StateStoreProvider
trait StateStoreProvider extends AnyRef
Trait representing a provider that provide StateStore instances representing versions of state data.
The life cycle of a provider and its provide stores are as follows.
- A StateStoreProvider is created in a executor for each unique StateStoreId when the first batch of a streaming query is executed on the executor. All subsequent batches reuse this provider instance until the query is stopped.
- Every batch of streaming data request a specific version of the state data by invoking
getStore(version) which returns an instance of StateStore through which the required
version of the data can be accessed. It is the responsible of the provider to populate
this store with context information like the schema of keys and values, etc.
If the checkpoint format version 2 is used, an additional argument checkpointID may be
provided as part of getStore(version, checkpointID). The provider needs to guarantee
that the loaded version is of this unique ID. It needs to load the version for this specific
ID from the checkpoint if needed.
- After the streaming query is stopped, the created provider instances are lazily disposed off.
- Alphabetic
- By Inheritance
- StateStoreProvider
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Abstract Value Members
- abstract def close(): Unit
Called when the provider instance is unloaded from the executor
- abstract def getStore(version: Long, stateStoreCkptId: Option[String] = None): StateStore
Return an instance of StateStore representing state data of the given version.
Return an instance of StateStore representing state data of the given version. If
stateStoreCkptIdis provided, the instance also needs to match the ID. - abstract def init(stateStoreId: StateStoreId, keySchema: StructType, valueSchema: StructType, keyStateEncoderSpec: KeyStateEncoderSpec, useColumnFamilies: Boolean, storeConfs: StateStoreConf, hadoopConf: Configuration, useMultipleValuesPerKey: Boolean = false, stateSchemaProvider: Option[StateSchemaProvider] = None): Unit
Initialize the provide with more contextual information from the SQL operator.
Initialize the provide with more contextual information from the SQL operator. This method will be called first after creating an instance of the StateStoreProvider by reflection.
- stateStoreId
Id of the versioned StateStores that this provider will generate
- keySchema
Schema of keys to be stored
- valueSchema
Schema of value to be stored
- useColumnFamilies
Whether the underlying state store uses a single or multiple column families
- storeConfs
Configurations used by the StateStores
- hadoopConf
Hadoop configuration that could be used by StateStore to save state data
- useMultipleValuesPerKey
Whether the underlying state store needs to support multiple values for a single key.
- abstract def stateStoreId: StateStoreId
Return the id of the StateStores this provider will generate.
Return the id of the StateStores this provider will generate. Should be the same as the one passed in init().
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- def doMaintenance(): Unit
Optional method for providers to allow for background maintenance (e.g.
Optional method for providers to allow for background maintenance (e.g. compactions)
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def getReadStore(version: Long, uniqueId: Option[String] = None): ReadStateStore
Return an instance of ReadStateStore representing state data of the given version and uniqueID if provided.
Return an instance of ReadStateStore representing state data of the given version and uniqueID if provided. By default it will return the same instance as getStore(version) but wrapped to prevent modification. Providers can override and return optimized version of ReadStateStore based on the fact the instance will be only used for reading.
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- def supportedCustomMetrics: Seq[StateStoreCustomMetric]
Optional custom metrics that the implementation may want to report.
Optional custom metrics that the implementation may want to report.
- Note
The StateStore objects created by this provider must report the same custom metrics (specifically, same names) through
StateStore.metrics.
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)