trait StateStore extends ReadStateStore
Base trait for a versioned key-value store which provides both read and write operations. Each
instance of a StateStore represents a specific version of state data, and such instances are
created through a StateStoreProvider.
Unlike ReadStateStore, abort method may not be called if the commit method succeeds
to commit the change. (hasCommitted returns true.) Otherwise, abort method will be called.
Implementation should deal with resource cleanup in both methods, and also need to guard with
double resource cleanup.
- Alphabetic
- By Inheritance
- StateStore
- ReadStateStore
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Abstract Value Members
- abstract def abort(): Unit
Abort all the updates that have been made to the store.
Abort all the updates that have been made to the store. Implementations should ensure that no more updates (puts, removes) can be after an abort in order to avoid incorrect usage.
- Definition Classes
- StateStore → ReadStateStore
- abstract def commit(): Long
Commit all the updates that have been made to the store, and return the new version.
Commit all the updates that have been made to the store, and return the new version. Implementations should ensure that no more updates (puts, removes) can be after a commit in order to avoid incorrect usage.
- abstract def createColFamilyIfAbsent(colFamilyName: String, keySchema: StructType, valueSchema: StructType, keyStateEncoderSpec: KeyStateEncoderSpec, useMultipleValuesPerKey: Boolean = false, isInternal: Boolean = false): Unit
Create column family with given name, if absent.
Create column family with given name, if absent.
- returns
column family ID
- abstract def get(key: UnsafeRow, colFamilyName: String = StateStore.DEFAULT_COL_FAMILY_NAME): UnsafeRow
Get the current value of a non-null key.
Get the current value of a non-null key.
- returns
a non-null row if the key exists in the store, otherwise null.
- Definition Classes
- ReadStateStore
- abstract def getStateStoreCheckpointInfo(): StateStoreCheckpointInfo
Return information on recently generated checkpoints The information should only be usable when checkpoint format version 2 is used and underlying state store supports it.
Return information on recently generated checkpoints The information should only be usable when checkpoint format version 2 is used and underlying state store supports it. If it is not the case, the method can return a dummy result. The result eventually won't be sent to the driver, but not all the stateful operator is able to figure out whether the function should be called to now. They would anyway call it and pass it to StatefulOperator.setStateStoreCheckpointInfo(), where it will be ignored.
- abstract def hasCommitted: Boolean
Whether all updates have been committed
- abstract def id: StateStoreId
Unique identifier of the store
Unique identifier of the store
- Definition Classes
- ReadStateStore
- abstract def iterator(colFamilyName: String = StateStore.DEFAULT_COL_FAMILY_NAME): StateStoreIterator[UnsafeRowPair]
Return an iterator containing all the key-value pairs in the StateStore.
Return an iterator containing all the key-value pairs in the StateStore. Implementations must ensure that updates (puts, removes) can be made while iterating over this iterator.
It is not required for implementations to ensure the iterator reflects all updates being performed after initialization of the iterator. Callers should perform all updates before calling this method if all updates should be visible in the returned iterator.
- Definition Classes
- StateStore → ReadStateStore
- abstract def merge(key: UnsafeRow, value: UnsafeRow, colFamilyName: String = StateStore.DEFAULT_COL_FAMILY_NAME): Unit
Merges the provided value with existing values of a non-null key.
Merges the provided value with existing values of a non-null key. If a existing value does not exist, this operation behaves as StateStore.put().
It is expected to throw exception if Spark calls this method without setting multipleValuesPerKey as true for the column family.
- abstract def metrics: StateStoreMetrics
Current metrics of the state store
- abstract def prefixScan(prefixKey: UnsafeRow, colFamilyName: String = StateStore.DEFAULT_COL_FAMILY_NAME): StateStoreIterator[UnsafeRowPair]
Return an iterator containing all the key-value pairs which are matched with the given prefix key.
Return an iterator containing all the key-value pairs which are matched with the given prefix key.
The operator will provide numColsPrefixKey greater than 0 in StateStoreProvider.init method if the operator needs to leverage the "prefix scan" feature. The schema of the prefix key should be same with the leftmost
numColsPrefixKeycolumns of the key schema.It is expected to throw exception if Spark calls this method without setting numColsPrefixKey to the greater than 0.
- Definition Classes
- ReadStateStore
- abstract def put(key: UnsafeRow, value: UnsafeRow, colFamilyName: String = StateStore.DEFAULT_COL_FAMILY_NAME): Unit
Put a new non-null value for a non-null key.
Put a new non-null value for a non-null key. Implementations must be aware that the UnsafeRows in the params can be reused, and must make copies of the data as needed for persistence.
- abstract def remove(key: UnsafeRow, colFamilyName: String = StateStore.DEFAULT_COL_FAMILY_NAME): Unit
Remove a single non-null key.
- abstract def removeColFamilyIfExists(colFamilyName: String): Boolean
Remove column family with given name, if present.
- abstract def valuesIterator(key: UnsafeRow, colFamilyName: String = StateStore.DEFAULT_COL_FAMILY_NAME): Iterator[UnsafeRow]
Provides an iterator containing all values of a non-null key.
Provides an iterator containing all values of a non-null key. If key does not exist, an empty iterator is returned. Implementations should make sure to return an empty iterator if the key does not exist.
It is expected to throw exception if Spark calls this method without setting multipleValuesPerKey as true for the column family.
- Definition Classes
- ReadStateStore
- abstract def version: Long
Version of the data in this store before committing updates.
Version of the data in this store before committing updates.
- Definition Classes
- ReadStateStore
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)