Packages

trait StateStore extends ReadStateStore

Base trait for a versioned key-value store which provides both read and write operations. Each instance of a StateStore represents a specific version of state data, and such instances are created through a StateStoreProvider.

Unlike ReadStateStore, abort method may not be called if the commit method succeeds to commit the change. (hasCommitted returns true.) Otherwise, abort method will be called. Implementation should deal with resource cleanup in both methods, and also need to guard with double resource cleanup.

Linear Supertypes
ReadStateStore, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. StateStore
  2. ReadStateStore
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Abstract Value Members

  1. abstract def abort(): Unit

    Abort all the updates that have been made to the store.

    Abort all the updates that have been made to the store. Implementations should ensure that no more updates (puts, removes) can be after an abort in order to avoid incorrect usage.

    Definition Classes
    StateStoreReadStateStore
  2. abstract def commit(): Long

    Commit all the updates that have been made to the store, and return the new version.

    Commit all the updates that have been made to the store, and return the new version. Implementations should ensure that no more updates (puts, removes) can be after a commit in order to avoid incorrect usage.

  3. abstract def createColFamilyIfAbsent(colFamilyName: String, keySchema: StructType, valueSchema: StructType, keyStateEncoderSpec: KeyStateEncoderSpec, useMultipleValuesPerKey: Boolean = false, isInternal: Boolean = false): Unit

    Create column family with given name, if absent.

    Create column family with given name, if absent.

    returns

    column family ID

  4. abstract def get(key: UnsafeRow, colFamilyName: String = StateStore.DEFAULT_COL_FAMILY_NAME): UnsafeRow

    Get the current value of a non-null key.

    Get the current value of a non-null key.

    returns

    a non-null row if the key exists in the store, otherwise null.

    Definition Classes
    ReadStateStore
  5. abstract def getStateStoreCheckpointInfo(): StateStoreCheckpointInfo

    Return information on recently generated checkpoints The information should only be usable when checkpoint format version 2 is used and underlying state store supports it.

    Return information on recently generated checkpoints The information should only be usable when checkpoint format version 2 is used and underlying state store supports it. If it is not the case, the method can return a dummy result. The result eventually won't be sent to the driver, but not all the stateful operator is able to figure out whether the function should be called to now. They would anyway call it and pass it to StatefulOperator.setStateStoreCheckpointInfo(), where it will be ignored.

  6. abstract def hasCommitted: Boolean

    Whether all updates have been committed

  7. abstract def id: StateStoreId

    Unique identifier of the store

    Unique identifier of the store

    Definition Classes
    ReadStateStore
  8. abstract def iterator(colFamilyName: String = StateStore.DEFAULT_COL_FAMILY_NAME): StateStoreIterator[UnsafeRowPair]

    Return an iterator containing all the key-value pairs in the StateStore.

    Return an iterator containing all the key-value pairs in the StateStore. Implementations must ensure that updates (puts, removes) can be made while iterating over this iterator.

    It is not required for implementations to ensure the iterator reflects all updates being performed after initialization of the iterator. Callers should perform all updates before calling this method if all updates should be visible in the returned iterator.

    Definition Classes
    StateStoreReadStateStore
  9. abstract def merge(key: UnsafeRow, value: UnsafeRow, colFamilyName: String = StateStore.DEFAULT_COL_FAMILY_NAME): Unit

    Merges the provided value with existing values of a non-null key.

    Merges the provided value with existing values of a non-null key. If a existing value does not exist, this operation behaves as StateStore.put().

    It is expected to throw exception if Spark calls this method without setting multipleValuesPerKey as true for the column family.

  10. abstract def metrics: StateStoreMetrics

    Current metrics of the state store

  11. abstract def prefixScan(prefixKey: UnsafeRow, colFamilyName: String = StateStore.DEFAULT_COL_FAMILY_NAME): StateStoreIterator[UnsafeRowPair]

    Return an iterator containing all the key-value pairs which are matched with the given prefix key.

    Return an iterator containing all the key-value pairs which are matched with the given prefix key.

    The operator will provide numColsPrefixKey greater than 0 in StateStoreProvider.init method if the operator needs to leverage the "prefix scan" feature. The schema of the prefix key should be same with the leftmost numColsPrefixKey columns of the key schema.

    It is expected to throw exception if Spark calls this method without setting numColsPrefixKey to the greater than 0.

    Definition Classes
    ReadStateStore
  12. abstract def put(key: UnsafeRow, value: UnsafeRow, colFamilyName: String = StateStore.DEFAULT_COL_FAMILY_NAME): Unit

    Put a new non-null value for a non-null key.

    Put a new non-null value for a non-null key. Implementations must be aware that the UnsafeRows in the params can be reused, and must make copies of the data as needed for persistence.

  13. abstract def remove(key: UnsafeRow, colFamilyName: String = StateStore.DEFAULT_COL_FAMILY_NAME): Unit

    Remove a single non-null key.

  14. abstract def removeColFamilyIfExists(colFamilyName: String): Boolean

    Remove column family with given name, if present.

  15. abstract def valuesIterator(key: UnsafeRow, colFamilyName: String = StateStore.DEFAULT_COL_FAMILY_NAME): Iterator[UnsafeRow]

    Provides an iterator containing all values of a non-null key.

    Provides an iterator containing all values of a non-null key. If key does not exist, an empty iterator is returned. Implementations should make sure to return an empty iterator if the key does not exist.

    It is expected to throw exception if Spark calls this method without setting multipleValuesPerKey as true for the column family.

    Definition Classes
    ReadStateStore
  16. abstract def version: Long

    Version of the data in this store before committing updates.

    Version of the data in this store before committing updates.

    Definition Classes
    ReadStateStore

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  8. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  9. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  10. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  11. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  12. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  13. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  14. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  15. def toString(): String
    Definition Classes
    AnyRef → Any
  16. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  17. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  18. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from ReadStateStore

Inherited from AnyRef

Inherited from Any

Ungrouped