Packages

object StateStore extends Logging

Companion object to StateStore that provides helper methods to create and retrieve stores by their unique ids. In addition, when a SparkContext is active (i.e. SparkEnv.get is not null), it also runs a periodic background task to do maintenance on the loaded stores. For each store, it uses the StateStoreCoordinator to ensure whether the current loaded instance of the store is the active instance. Accordingly, it either keeps it loaded and performs maintenance, or unloads the store.

Linear Supertypes
Logging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. StateStore
  2. Logging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Type Members

  1. implicit class LogStringContext extends AnyRef
    Definition Classes
    Logging
  2. class MaintenanceTask extends AnyRef

    Runs the task periodically and bubbles any exceptions that it encounters.

    Runs the task periodically and bubbles any exceptions that it encounters.

    Note: exceptions in the maintenance thread pool are caught and logged; the associated StateStoreProvider is also unloaded. Any exception that happens in the MaintenanceTask is indeed exceptional and thus we let it propagate.

  3. class MaintenanceThreadPool extends AnyRef

    Thread Pool that runs maintenance on partitions that are scheduled by MaintenanceTask periodically

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. val DEFAULT_COL_FAMILY_NAME: String
  5. def MDC(key: LogKey, value: Any): MDC
    Attributes
    protected
    Definition Classes
    Logging
  6. val PARTITION_ID_TO_CHECK_SCHEMA: Int
  7. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  8. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  9. def doMaintenanceAndUnload(storeProviderId: StateStoreProviderId): Unit

    Runs maintenance and then unload a state store provider

  10. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  11. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  12. def get(storeProviderId: StateStoreProviderId, keySchema: StructType, valueSchema: StructType, keyStateEncoderSpec: KeyStateEncoderSpec, version: Long, stateStoreCkptId: Option[String], stateSchemaBroadcast: Option[StateSchemaBroadcast], useColumnFamilies: Boolean, storeConf: StateStoreConf, hadoopConf: Configuration, useMultipleValuesPerKey: Boolean = false): StateStore

    Get or create a store associated with the id.

  13. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  14. def getReadOnly(storeProviderId: StateStoreProviderId, keySchema: StructType, valueSchema: StructType, keyStateEncoderSpec: KeyStateEncoderSpec, version: Long, stateStoreCkptId: Option[String], stateSchemaBroadcast: Option[StateSchemaBroadcast], useColumnFamilies: Boolean, storeConf: StateStoreConf, hadoopConf: Configuration, useMultipleValuesPerKey: Boolean = false): ReadStateStore

    Get or create a read-only store associated with the id.

  15. def getWriteStore(readStore: ReadStateStore, storeProviderId: StateStoreProviderId, keySchema: StructType, valueSchema: StructType, keyStateEncoderSpec: KeyStateEncoderSpec, version: Long, stateStoreCkptId: Option[String], stateSchemaBroadcast: Option[StateSchemaBroadcast], useColumnFamilies: Boolean, storeConf: StateStoreConf, hadoopConf: Configuration, useMultipleValuesPerKey: Boolean = false): StateStore

    Converts an existing read-only state store to a writable state store.

    Converts an existing read-only state store to a writable state store.

    This method provides an optimization for stateful operations that need to both read and update state within the same task. Instead of opening separate read and write instances (which may cause resource contention or duplication), this method reuses the already loaded read store and transforms it into a writable store.

    The optimization is particularly valuable for state stores with expensive initialization costs or limited concurrency capabilities (like RocksDB). It eliminates redundant loading of the same state data and reduces resource usage.

    readStore

    The existing read-only state store to convert to a writable store

    storeProviderId

    Unique identifier for the state store provider

    keySchema

    Schema of the state store keys

    valueSchema

    Schema of the state store values

    keyStateEncoderSpec

    Specification for encoding the state keys

    version

    The version of the state store (must match the read store's version)

    stateStoreCkptId

    Optional checkpoint identifier for the state store

    stateSchemaBroadcast

    Optional broadcast of the state schema

    useColumnFamilies

    Whether to use column families in the state store

    storeConf

    Configuration for the state store

    hadoopConf

    Hadoop configuration

    useMultipleValuesPerKey

    Whether the store supports multiple values per key

    returns

    A writable StateStore instance that can be used to update and commit changes

    Exceptions thrown

    SparkException If the store cannot be loaded or if there's insufficient memory

  16. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  17. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  18. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  19. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  20. def isLoaded(storeProviderId: StateStoreProviderId): Boolean

    Whether a state store provider is loaded or not

  21. def isMaintenanceRunning: Boolean

    Check if maintenance thread is running and scheduled future is not done

  22. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  23. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  24. def logBasedOnLevel(level: Level)(f: => MessageWithContext): Unit
    Attributes
    protected
    Definition Classes
    Logging
  25. def logDebug(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  26. def logDebug(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  27. def logDebug(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  28. def logDebug(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  29. def logError(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  30. def logError(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  31. def logError(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  32. def logError(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  33. def logInfo(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  34. def logInfo(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  35. def logInfo(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  36. def logInfo(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  37. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  38. def logTrace(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  39. def logTrace(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  40. def logTrace(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  41. def logTrace(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  42. def logWarning(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  43. def logWarning(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  44. def logWarning(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  45. def logWarning(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  46. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  47. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  48. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  49. def removeFromLoadedProvidersAndClose(storeProviderId: StateStoreProviderId, alreadyRemovedProvider: Option[StateStoreProvider] = None): Unit

    Unload a state store provider.

    Unload a state store provider. If alreadyRemovedFromLoadedProviders is None, provider will be removed from loadedProviders and closed. If alreadyRemovedFromLoadedProviders is Some, provider will be closed using passed in provider. WARNING: CAN ONLY BE CALLED FROM MAINTENANCE THREAD!

  50. def reportCommitToCoordinator(version: Long, stateStoreId: StateStoreId, hadoopConf: Configuration): Unit

    Reports to the coordinator that a StateStore has committed

  51. def stop(): Unit

    Unload and stop all state store providers

  52. def stopMaintenanceTask(): Unit

    Stop maintenance thread and reset the maintenance task

  53. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  54. def toString(): String
    Definition Classes
    AnyRef → Any
  55. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  56. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  57. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  58. def withLogContext(context: Map[String, String])(body: => Unit): Unit
    Attributes
    protected
    Definition Classes
    Logging

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped