class StateStoreConf extends Serializable
A class that contains configuration parameters for StateStores.
- Alphabetic
- By Inheritance
- StateStoreConf
- Serializable
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- val checkpointFileChecksumEnabled: Boolean
Whether file checksum generation and verification is enabled.
- val checkpointFileChecksumSkipCreationIfFileMissingChecksum: Boolean
Whether to skip checksum creation if file missing checksum.
Whether to skip checksum creation if file missing checksum.
Consider the case using STATE_STORE_CHECKPOINT_FORMAT_VERSION = 1 when a batch fails but state files are written. If on the next run, we try to upload both a new state file and a file checksum, the file could fail to be uploaded but the file checksum is uploaded successfully. This would lead to a situation where the old file could be loaded and compared with the new file checksum, which would fail the checksum verification. This issue does not happen when STATE_STORE_CHECKPOINT_FORMAT_VERSION = 2 since each batch run unique ids will be created.
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- val commitValidationEnabled: Boolean
Whether to validate StateStore commits for ForeachBatch sinks to ensure all partitions are processed.
Whether to validate StateStore commits for ForeachBatch sinks to ensure all partitions are processed. This helps detect incomplete processing due to operations like show() or limit().
- val compressionCodec: String
The compression codec used to compress delta and snapshot files.
- val enableStateStoreCheckpointIds: Boolean
When creating new state store checkpoint, which format version to use.
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- val extraOptions: Map[String, String]
- val formatValidationCheckValue: Boolean
Whether to validate the value side.
Whether to validate the value side. This config is applied to both validators as below:
- whether to validate the value format when the format validation is enabled. - whether to validate the value schema when the state schema check is enabled.
- val formatValidationEnabled: Boolean
Whether validate the underlying format or not.
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- val maintenanceInterval: Long
The interval of maintenance tasks.
- val maxVersionsToDeletePerMaintenance: Int
Maximum number of versions to delete per maintenance operation
- val maxVersionsToRetainInMemory: Int
Maximum count of versions a State Store implementation should retain in memory
- val minDeltasForSnapshot: Int
Minimum number of delta files in a chain after which HDFSBackedStateStore will consider generating a snapshot.
- val minVersionsToDelete: Long
Minimum number of stale checkpoint versions that need to be present in the DFS checkpoint directory for old state checkpoint version deletion to be invoked.
Minimum number of stale checkpoint versions that need to be present in the DFS checkpoint directory for old state checkpoint version deletion to be invoked. This is to amortize the cost of discovering and deleting old checkpoint versions.
- val minVersionsToRetain: Int
Minimum versions a State Store implementation should retain to allow rollbacks
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- val numStateStoreMaintenanceThreads: Int
Size of MaintenanceThreadPool to perform maintenance tasks for StateStore
- val providerClass: String
Optional fully qualified name of the subclass of StateStoreProvider managing state data.
Optional fully qualified name of the subclass of StateStoreProvider managing state data. That is, the implementation of the State Store to use.
- val reportSnapshotUploadLag: Boolean
Whether the coordinator is reporting state stores trailing behind in snapshot uploads.
- val skipNullsForStreamStreamJoins: Boolean
Whether to skip null values for hash based stream-stream joins.
- val sqlConfs: Map[String, String]
Additional configurations related to state store.
Additional configurations related to state store. This will capture all configs in SQLConf that start with
spark.sql.streaming.stateStore. - val stateSchemaCheckEnabled: Boolean
whether to validate state schema during query run.
- val stateStoreCheckpointFormatVersion: Int
The version of the state store checkpoint format.
- val stateStoreEncodingFormat: String
The interval of maintenance tasks.
- val stateStoreMaintenanceProcessingTimeout: Long
- val stateStoreMaintenanceShutdownTimeout: Long
Timeout for state store maintenance operations to complete on shutdown
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- val unloadOnCommit: Boolean
Whether to unload the store on task completion.
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)