Type Members

1.  case class AcquiredThreadInfo(threadRef: WeakReference[Thread] = new WeakReference[Thread](Thread.currentThread()), tc: TaskContext = TaskContext.get()) extends Product with Serializable
2.  case class AvroEncoder(keySerializer: AvroSerializer, keyDeserializer: AvroDeserializer, valueSerializer: AvroSerializer, valueDeserializer: AvroDeserializer, suffixKeySerializer: Option[AvroSerializer] = None, suffixKeyDeserializer: Option[AvroDeserializer] = None) extends Serializable with Product

   An Avro-based encoder used for serializing between UnsafeRow and Avro byte arrays in RocksDB state stores.

   An Avro-based encoder used for serializing between UnsafeRow and Avro byte arrays in RocksDB state stores.

   This encoder is primarily utilized by RocksDBStateStoreProvider and RocksDBStateEncoder to handle serialization and deserialization of state store data.

   keySerializer

   Serializer for converting state store keys to Avro format

   keyDeserializer

   Deserializer for converting Avro-encoded keys back to UnsafeRow

   valueSerializer

   Serializer for converting state store values to Avro format

   valueDeserializer

   Deserializer for converting Avro-encoded values back to UnsafeRow

   suffixKeySerializer

   Optional serializer for handling suffix keys in Avro format

   suffixKeyDeserializer

   Optional deserializer for converting Avro-encoded suffix keys back to UnsafeRow

3.  class AvroStateEncoder extends RocksDBDataEncoder with Logging

   Encoder that uses Avro for serializing state store data with schema evolution support.

   Encoder that uses Avro for serializing state store data with schema evolution support. The encoded format varies depending on the key type and whether it's a key or value:

   For prefix and range scan keys:

   --prefix---	--schemaId (2 bytes)--	--remainingKeyBytes (avro-encoded)--

   where: - prefix: Variable length prefix for scan operations - schemaId: 2 byte short integer identifying the schema version - remainingKeyBytes: Avro-encoded remaining key data

   For no-prefix keys and values:

   --schemaId (2 bytes)--	--avroEncodedBytes--

   where: - schemaId: 2 byte short integer identifying the schema version - avroEncodedBytes: Variable length Avro-encoded data

   The schema ID allows the state store to identify which schema version was used to encode the data, enabling proper decoding even when schemas have evolved over time.

4.  abstract class BaseStateStoreRDD[T, U] extends RDD[U]
5.  class ByteArrayPair extends AnyRef

   Mutable and reusable pair of byte arrays

6.  case class ColumnFamilyInfo(cfId: Short, isInternal: Boolean) extends Product with Serializable

   Case class to keep track of column family info within checkpoint metadata.

   Case class to keep track of column family info within checkpoint metadata.

   cfId

   - virtual column family id

   isInternal

   - whether the column family is internal or not

7.  trait DataEncoder extends AnyRef

   The DataEncoder can encode UnsafeRows into raw bytes in two ways:

   The DataEncoder can encode UnsafeRows into raw bytes in two ways:

   • Using the direct byte layout of the UnsafeRow
   • Converting the UnsafeRow into an Avro row, and encoding that In both of these cases, the raw bytes that are written into RockDB have headers, footers and other metadata, but they also have data that is provided by the callers. The metadata in each row does not need to be written as Avro or UnsafeRow, but the actual data provided by the caller does. The classes that use this trait require specialized partial encoding which makes them much easier to cache and use, which is why each DataEncoder deals with multiple schemas.
8.  class HDFSBackedStateStoreChangeDataReader extends StateStoreChangeDataReader

   StateStoreChangeDataReader implementation for HDFSBackedStateStoreProvider

9.  trait HDFSBackedStateStoreMap extends AnyRef
10.  class InMemoryStateSchemaProvider extends StateSchemaProvider with Logging
11.  sealed trait KeyStateEncoderSpec extends AnyRef
12.  case class LineageItem(version: Long, checkpointUniqueId: String) extends Product with Serializable

    Class for lineage item for checkpoint format V2.

13.  class MultiValuedStateEncoder extends RocksDBValueStateEncoder with Logging

    Supports encoding multiple values per key in RocksDB.

    Supports encoding multiple values per key in RocksDB. A single value is encoded in the format below, where first value is number of bytes in actual encodedUnsafeRow followed by the encoded value itself.

    ---size(bytes)--	--unsafeRowEncodedBytes--

    Multiple values are separated by a delimiter character.

    This encoder supports RocksDB StringAppendOperator merge operator. Values encoded can be merged in RocksDB using merge operation, and all merged values can be read using decodeValues operation.

14.  class NoPrefixHDFSBackedStateStoreMap extends HDFSBackedStateStoreMap
15.  class NoPrefixKeyStateEncoder extends RocksDBKeyStateEncoder with Logging

    RocksDB Key Encoder for UnsafeRow that does not support prefix key scan.

    RocksDB Key Encoder for UnsafeRow that does not support prefix key scan.

    Encodes/decodes UnsafeRows to versioned byte arrays. It uses the first byte of the generated byte array to store the version the describes how the row is encoded in the rest of the byte array. Currently, the default version is 0,

    VERSION 0: [ VERSION (1 byte) | ROW (N bytes) ] The bytes of a UnsafeRow is written unmodified to starting from offset 1 (offset 0 is the version byte of value 0). That is, if the unsafe row has N bytes, then the generated array byte will be N+1 bytes.

16.  case class NoPrefixKeyStateEncoderSpec(keySchema: StructType) extends KeyStateEncoderSpec with Product with Serializable
17.  trait OperatorInfo extends AnyRef

    Information about a stateful operator.

18.  case class OperatorInfoV1(operatorId: Long, operatorName: String) extends OperatorInfo with Product with Serializable
19.  trait OperatorStateMetadata extends AnyRef
20.  case class OperatorStateMetadataV1(operatorInfo: OperatorInfoV1, stateStoreInfo: Array[StateStoreMetadataV1]) extends OperatorStateMetadata with Product with Serializable
21.  class OperatorStateMetadataV1Reader extends OperatorStateMetadataReader

    Read OperatorStateMetadata from the state checkpoint directory.

    Read OperatorStateMetadata from the state checkpoint directory. This class will only be used to read OperatorStateMetadataV1. OperatorStateMetadataV2 will be read by the OperatorStateMetadataLog.

22.  class OperatorStateMetadataV1Writer extends OperatorStateMetadataWriter with Logging

    Write OperatorStateMetadata into the state checkpoint directory.

23.  case class OperatorStateMetadataV2(operatorInfo: OperatorInfoV1, stateStoreInfo: Array[StateStoreMetadataV2], operatorPropertiesJson: String) extends OperatorStateMetadata with Product with Serializable
24.  class OperatorStateMetadataV2FileManager extends Logging

    A helper class to manage the metadata files for the operator state checkpoint.

    A helper class to manage the metadata files for the operator state checkpoint. This class is used to manage the metadata files for OperatorStateMetadataV2, and provides utils to purge the oldest files such that we only keep the metadata files for which a commit log is present

25.  class OperatorStateMetadataV2Reader extends OperatorStateMetadataReader
26.  class OperatorStateMetadataV2Writer extends OperatorStateMetadataWriter
27.  class PrefixKeyScanStateEncoder extends RocksDBKeyStateEncoder with Logging

    RocksDB Key Encoder for UnsafeRow that supports prefix scan

28.  case class PrefixKeyScanStateEncoderSpec(keySchema: StructType, numColsPrefixKey: Int) extends KeyStateEncoderSpec with Product with Serializable
29.  class PrefixScannableHDFSBackedStateStoreMap extends HDFSBackedStateStoreMap
30.  class RangeKeyScanStateEncoder extends RocksDBKeyStateEncoder with Logging

    RocksDB Key Encoder for UnsafeRow that supports range scan for fixed size fields

    RocksDB Key Encoder for UnsafeRow that supports range scan for fixed size fields

    To encode a row for range scan, we first project the orderingOrdinals from the oridinal UnsafeRow into another UnsafeRow; we then rewrite that new UnsafeRow's fields in BIG_ENDIAN to allow for scanning keys in sorted order using the byte-wise comparison method that RocksDB uses.

    Then, for the rest of the fields, we project those into another UnsafeRow. We then effectively join these two UnsafeRows together, and finally take those bytes to get the resulting row.

    We cannot support variable sized fields in the range scan because the UnsafeRow format stores variable sized fields as offset and length pointers to the actual values, thereby changing the required ordering.

    Note that we also support "null" values being passed for these fixed size fields. We prepend a single byte to indicate whether the column value is null or not. We cannot change the nullability on the UnsafeRow itself as the expected ordering would change if non-first columns are marked as null. If the first col is null, those entries will appear last in the iterator. If non-first columns are null, ordering based on the previous columns will still be honored. For rows with null column values, ordering for subsequent columns will also be maintained within those set of rows. We use the same byte to also encode whether the value is negative or not. For negative float/double values, we flip all the bits to ensure the right lexicographical ordering. For the rationale around this, please check the link here: https://en.wikipedia.org/wiki/IEEE_754#Design_rationale

31.  case class RangeKeyScanStateEncoderSpec(keySchema: StructType, orderingOrdinals: Seq[Int]) extends KeyStateEncoderSpec with Product with Serializable

    Encodes rows so that they can be range-scanned based on orderingOrdinals

32.  trait ReadStateStore extends AnyRef

    Base trait for a versioned key-value store which provides read operations.

    Base trait for a versioned key-value store which provides read operations. Each instance of a ReadStateStore represents a specific version of state data, and such instances are created through a StateStoreProvider.

    abort method will be called when the task is completed - please clean up the resources in the method.

    IMPLEMENTATION NOTES: * The implementation can throw exception on calling prefixScan method if the functionality is not supported yet from the implementation. Note that some stateful operations would not work on disabling prefixScan functionality.

33.  class ReadStateStoreRDD[T, U] extends BaseStateStoreRDD[T, U]

    An RDD that allows computations to be executed against ReadStateStores.

    An RDD that allows computations to be executed against ReadStateStores. It uses the StateStoreCoordinator to get the locations of loaded state stores and use that as the preferred locations.

34.  class RocksDB extends Logging

    Class representing a RocksDB instance that checkpoints version of data to DFS.

    Class representing a RocksDB instance that checkpoints version of data to DFS. After a set of updates, a new version can be committed by calling commit(). Any past version can be loaded by calling load(version).

    Note

    This class is not thread-safe, so use it only from one thread.

    See also

    RocksDBFileManager to see how the files are laid out in local disk and DFS.

35.  case class RocksDBCheckpointMetadata(sstFiles: Seq[RocksDBSstFile], logFiles: Seq[RocksDBLogFile], numKeys: Long, numInternalKeys: Long, columnFamilyMapping: Option[Map[String, Short]] = None, columnFamilyTypeMap: Option[Map[String, Boolean]] = None, maxColumnFamilyId: Option[Short] = None) extends Product with Serializable

    Classes to represent metadata of checkpoints saved to DFS.

    Classes to represent metadata of checkpoints saved to DFS. Since this is converted to JSON, any changes to this MUST be backward-compatible.

36.  case class RocksDBConf(minVersionsToRetain: Int, minVersionsToDelete: Long, minDeltasForSnapshot: Int, compactOnCommit: Boolean, enableChangelogCheckpointing: Boolean, blockSizeKB: Long, blockCacheSizeMB: Long, lockAcquireTimeoutMs: Long, resetStatsOnLoad: Boolean, formatVersion: Int, trackTotalNumberOfRows: Boolean, maxOpenFiles: Int, writeBufferSizeMB: Long, maxWriteBufferNumber: Int, boundedMemoryUsage: Boolean, totalMemoryUsageMB: Long, writeBufferCacheRatio: Double, highPriorityPoolRatio: Double, compressionCodec: String, allowFAllocate: Boolean, compression: String) extends Product with Serializable

    Configurations for optimizing RocksDB

    Configurations for optimizing RocksDB

    compactOnCommit

    Whether to compact RocksDB data before commit / checkpointing

37.  abstract class RocksDBDataEncoder extends DataEncoder
38.  class RocksDBFileManager extends Logging

    Class responsible for syncing RocksDB checkpoint files from local disk to DFS.

    Class responsible for syncing RocksDB checkpoint files from local disk to DFS. For each version, checkpoint is saved in specific directory structure that allows successive versions to reuse to SST data files and archived log files. This allows each commit to be incremental, only new SST files and archived log files generated by RocksDB will be uploaded. The directory structures on local disk and in DFS are as follows.

    Local checkpoint dir structure ------------------------------ RocksDB generates a bunch of files in the local checkpoint directory. The most important among them are the SST files; they are the actual log structured data files. Rest of the files contain the metadata necessary for RocksDB to read the SST files and start from the checkpoint. Note that the SST files are hard links to files in the RocksDB's working directory, and therefore successive checkpoints can share some of the SST files. So these SST files have to be copied to DFS in shared directory such that different committed versions can save them.

    We consider both SST files and archived log files as immutable files which can be shared between different checkpoints.

    localCheckpointDir | +-- OPTIONS-000005 +-- MANIFEST-000008 +-- CURRENT +-- 00007.sst +-- 00011.sst +-- archive | +-- 00008.log | +-- 00013.log ...

    DFS directory structure after saving to DFS as version 10 ----------------------------------------------------------- The SST and archived log files are given unique file names and copied to the shared subdirectory. Every version maintains a mapping of local immutable file name to the unique file name in DFS. This mapping is saved in a JSON file (named metadata), which is zipped along with other checkpoint files into a single file [version].zip.

    dfsRootDir | +-- SSTs | +-- 00007-[uuid1].sst | +-- 00011-[uuid2].sst +-- logs | +-- 00008-[uuid3].log | +-- 00013-[uuid4].log +-- 10.zip | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst, and the mapping between 00008.log and [uuid3].log | +-- OPTIONS-000005 | +-- MANIFEST-000008 | +-- CURRENT | ... | +-- 9.zip +-- 8.zip ...

    Note the following. - Each [version].zip is a complete description of all the data and metadata needed to recover a RocksDB instance at the corresponding version. The SST files and log files are not included in the zip files, they can be shared cross different versions. This is unlike the [version].delta files of HDFSBackedStateStore where previous delta files needs to be read to be recovered. - This is safe wrt speculatively executed tasks running concurrently in different executors as each task would upload a different copy of the generated immutable files and atomically update the [version].zip. - Immutable files are identified uniquely based on their file name and file size. - Immutable files can be reused only across adjacent checkpoints/versions. - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a different thread than the task thread saving files.

39.  case class RocksDBFileManagerMetrics(filesCopied: Long, bytesCopied: Long, filesReused: Long, zipFileBytesUncompressed: Option[Long] = None) extends Product with Serializable

    Metrics regarding RocksDB file sync between local and DFS.

40.  class RocksDBFileMapping extends AnyRef
41.  sealed trait RocksDBImmutableFile extends AnyRef

    A RocksDBImmutableFile maintains a mapping between a local RocksDB file name and the name of its copy on DFS.

    A RocksDBImmutableFile maintains a mapping between a local RocksDB file name and the name of its copy on DFS. Since these files are immutable, their DFS copies can be reused.

42.  sealed trait RocksDBKeyStateEncoder extends AnyRef
43.  case class RocksDBMetrics(numCommittedKeys: Long, numUncommittedKeys: Long, numInternalKeys: Long, totalMemUsageBytes: Long, pinnedBlocksMemUsage: Long, totalSSTFilesBytes: Long, nativeOpsHistograms: Map[String, RocksDBNativeHistogram], lastCommitLatencyMs: Map[String, Long], filesCopied: Long, bytesCopied: Long, filesReused: Long, zipFileBytesUncompressed: Option[Long], nativeOpsMetrics: Map[String, Long]) extends Product with Serializable

    Class to represent stats from each commit.

44.  case class RocksDBNativeHistogram(sum: Long, avg: Double, stddev: Double, median: Double, p95: Double, p99: Double, count: Long) extends Product with Serializable

    Class to wrap RocksDB's native histogram

45.  sealed abstract class RocksDBOpType extends AnyRef
46.  case class RocksDBSnapshotFile(immutableFile: RocksDBImmutableFile, isUploaded: Boolean) extends Product with Serializable
47.  class RocksDBStateStoreChangeDataReader extends StateStoreChangeDataReader

    StateStoreChangeDataReader implementation for RocksDBStateStoreProvider

48.  sealed trait RocksDBValueStateEncoder extends AnyRef
49.  case class RocksDBVersionSnapshotInfo(version: Long, dfsFilesUUID: String) extends Product with Serializable
50.  class SingleValueStateEncoder extends RocksDBValueStateEncoder with Logging

    RocksDB Value Encoder for UnsafeRow that only supports single value.

    RocksDB Value Encoder for UnsafeRow that only supports single value.

    Encodes/decodes UnsafeRows to versioned byte arrays. It uses the first byte of the generated byte array to store the version the describes how the row is encoded in the rest of the byte array. Currently, the default version is 0,

    VERSION 0: [ VERSION (1 byte) | ROW (N bytes) ] The bytes of a UnsafeRow is written unmodified to starting from offset 1 (offset 0 is the version byte of value 0). That is, if the unsafe row has N bytes, then the generated array byte will be N+1 bytes.

51.  final class StateMessage extends AnyRef
52.  case class StateRowEncoderCacheKey(queryRunId: String, operatorId: Long, partitionId: Int, stateStoreName: String, colFamilyName: String) extends Product with Serializable
53.  case class StateSchemaBroadcast(broadcast: Broadcast[StateSchemaMetadata]) extends Logging with StateSchemaProvider with Product with Serializable

    Broadcasts schema metadata information for stateful operators in a streaming query.

    Broadcasts schema metadata information for stateful operators in a streaming query.

    This class provides a way to distribute schema evolution information to all executors via Spark's broadcast mechanism. Each stateful operator in a streaming query maintains its own instance of this class to track schema versions and evolution.

    broadcast

    Spark broadcast variable containing the schema metadata

54.  class StateSchemaCompatibilityChecker extends Logging
55.  case class StateSchemaFormatV3(stateStoreColFamilySchema: List[StateStoreColFamilySchema]) extends Product with Serializable

    Helper classes for reading/writing state schema.

56.  case class StateSchemaIdRow(schemaId: Short, bytes: Array[Byte]) extends Product with Serializable

    Represents a row of state data along with its schema version.

    Represents a row of state data along with its schema version. Used during state storage operations to track which schema version was used to encode the data, enabling proper decoding even when schemas have evolved.

    schemaId

    The version identifier for the schema that was used to encode this row. This could be either a key schema ID or value schema ID depending on context.

    bytes

    The actual encoded data bytes for this row. When using Avro encoding, these bytes contain the Avro-serialized data. For UnsafeRow encoding, these contain the binary-encoded row data.

57.  case class StateSchemaInfo(keySchemaId: Short, valueSchemaId: Short) extends Product with Serializable

    Contains schema version information for both key and value schemas in a state store.

    Contains schema version information for both key and value schemas in a state store. This information is used to support schema evolution, allowing state schemas to be modified over time while maintaining compatibility with existing state data.

    keySchemaId

    A unique identifier for the version of the key schema. Used to track and handle changes to the key schema structure.

    valueSchemaId

    A unique identifier for the version of the value schema. Used to track and handle changes to the value schema structure.

58.  case class StateSchemaMetadata(activeSchemas: Map[StateSchemaMetadataKey, StateSchemaMetadataValue]) extends Product with Serializable

    Contains schema evolution metadata for a stateful operator.

    Contains schema evolution metadata for a stateful operator.

    activeSchemas

    Map of all active schema versions, keyed by column family and schema ID. This includes both the current schema and any previous schemas that may still exist in the state store.

59.  case class StateSchemaMetadataKey(colFamilyName: String, schemaId: Short, isKey: Boolean) extends Product with Serializable

    Composite key for looking up schema metadata, combining column family and schema version.

    Composite key for looking up schema metadata, combining column family and schema version.

    colFamilyName

    Name of the RocksDB column family this schema applies to

    schemaId

    Version identifier for this schema

60.  case class StateSchemaMetadataValue(sqlSchema: StructType, avroSchema: Schema) extends Product with Serializable

    Contains both SQL and Avro representations of a schema version.

    Contains both SQL and Avro representations of a schema version.

    The SQL schema represents the logical structure while the Avro schema is used for evolution compatibility checking and serialization.

    sqlSchema

    The Spark SQL schema definition

    avroSchema

    The equivalent Avro schema used for compatibility checking

61.  trait StateSchemaProvider extends Serializable
62.  case class StateSchemaValidationResult(evolvedSchema: Boolean, schemaPath: String) extends Product with Serializable
63.  trait StateStore extends ReadStateStore

    Base trait for a versioned key-value store which provides both read and write operations.

    Base trait for a versioned key-value store which provides both read and write operations. Each instance of a StateStore represents a specific version of state data, and such instances are created through a StateStoreProvider.

    Unlike ReadStateStore, abort method may not be called if the commit method succeeds to commit the change. (hasCommitted returns true.) Otherwise, abort method will be called. Implementation should deal with resource cleanup in both methods, and also need to guard with double resource cleanup.

64.  class StateStoreCannotCreateColumnFamilyWithReservedChars extends SparkUnsupportedOperationException
65.  class StateStoreCannotUseColumnFamilyWithInvalidName extends SparkUnsupportedOperationException
66.  abstract class StateStoreChangeDataReader extends NextIterator[(state.RecordType.Value, UnsafeRow, UnsafeRow, Long)] with Logging

    Base class representing a iterator that iterates over a range of changelog files in a state store.

    Base class representing a iterator that iterates over a range of changelog files in a state store. In each iteration, it will return a tuple of (changeType: RecordType, nested key: UnsafeRow, nested value: UnsafeRow, batchId: Long)

67.  abstract class StateStoreChangelogReader extends NextIterator[(state.RecordType.Value, Array[Byte], Array[Byte])] with Logging

    Base class for state store changelog reader

68.  class StateStoreChangelogReaderFactory extends Logging

    A factory class for constructing state store readers by reading the first line of the change log file, which stores the version.

    A factory class for constructing state store readers by reading the first line of the change log file, which stores the version. Note that for changelog version 1, there is no version written.

69.  class StateStoreChangelogReaderV1 extends StateStoreChangelogReader

    Read an iterator of change record from the changelog file.

    Read an iterator of change record from the changelog file. A record is represented by tuple(recordType: RecordType.Value, key: Array[Byte], value: Array[Byte]) A put record is returned as a tuple(recordType, key, value) A delete record is return as a tuple(recordType, key, null)

70.  class StateStoreChangelogReaderV2 extends StateStoreChangelogReader

    Read an iterator of change record from the changelog file.

    Read an iterator of change record from the changelog file. A record is represented by tuple(recordType: RecordType.Value, key: Array[Byte], value: Array[Byte]) A put or merge record is returned as a tuple(recordType, key, value) A delete record is return as a tuple(recordType, key, null)

71.  class StateStoreChangelogReaderV3 extends StateStoreChangelogReaderV1

    Read an iterator of change record from the changelog file.

    Read an iterator of change record from the changelog file. A record is represented by tuple(recordType: RecordType.Value, key: Array[Byte], value: Array[Byte]) A put record is returned as a tuple(recordType, key, value) A delete record is return as a tuple(recordType, key, null) V3 is a extension of V1 for reading changelogs with version in the first line and lineage in the second line.

72.  class StateStoreChangelogReaderV4 extends StateStoreChangelogReaderV2

    Read an iterator of change record from the changelog file.

    Read an iterator of change record from the changelog file. A record is represented by tuple(recordType: RecordType.Value, key: Array[Byte], value: Array[Byte]) A put or merge record is returned as a tuple(recordType, key, value) A delete record is return as a tuple(recordType, key, null) V4 is a extension of V2 for reading changelogs with version in the first line and lineage in the second line.

73.  abstract class StateStoreChangelogWriter extends Logging

    Base class for state store changelog writer

74.  class StateStoreChangelogWriterV1 extends StateStoreChangelogWriter

    Write changes to the key value state store instance to a changelog file.

    Write changes to the key value state store instance to a changelog file. There are 2 types of records, put and delete. A put record is written as: | key length | key content | value length | value content | A delete record is written as: | key length | key content | -1 | Write an Int -1 to signal the end of file. The overall changelog format is: | put record | delete record | ... | put record | -1 |

75.  class StateStoreChangelogWriterV2 extends StateStoreChangelogWriter

    Write changes to the key value state store instance to a changelog file.

    Write changes to the key value state store instance to a changelog file. There are 3 types of data records, put, merge and delete. A put record or merge record is written as: | record type | key length | key content | value length | value content | -1 | A delete record is written as: | record type | key length | key content | -1 Write an EOF_RECORD to signal the end of file. The overall changelog format is: version | put record | delete record | ... | put record | eof record |

76.  class StateStoreChangelogWriterV3 extends StateStoreChangelogWriterV1

    Write changes to the key value state store instance to a changelog file.

    Write changes to the key value state store instance to a changelog file. There are 2 types of records, put and delete. A put record is written as: | key length | key content | value length | value content | A delete record is written as: | key length | key content | -1 | Write an Int -1 to signal the end of file. The overall changelog format is: | put record | delete record | ... | put record | -1 | V3 is a extension of V1 for writing changelogs with version in the first line and lineage in the second line.

77.  class StateStoreChangelogWriterV4 extends StateStoreChangelogWriterV2

    Write changes to the key value state store instance to a changelog file.

    Write changes to the key value state store instance to a changelog file. There are 3 types of data records, put, merge and delete. A put record or merge record is written as: | record type | key length | key content | value length | value content | -1 | A delete record is written as: | record type | key length | key content | -1 Write an EOF_RECORD to signal the end of file. The overall changelog format is: version | put record | delete record | ... | put record | eof record | V4 is a extension of V2 for writing changelogs with version in the first line and lineage in the second line.

78.  case class StateStoreCheckpointInfo(partitionId: Int, batchVersion: Long, stateStoreCkptId: Option[String], baseStateStoreCkptId: Option[String]) extends Product with Serializable

    State store checkpoint information, used to pass checkpointing information from executors to the driver after execution.

    State store checkpoint information, used to pass checkpointing information from executors to the driver after execution.

    stateStoreCkptId

    The checkpoint ID for a checkpoint at batchVersion. This is used to identify the checkpoint

    baseStateStoreCkptId

    The checkpoint ID for batchVersion - 1, that is used to finish this batch. This is used to validate the batch is processed based on the correct checkpoint.

79.  case class StateStoreColFamilySchema(colFamilyName: String, keySchemaId: Short, keySchema: StructType, valueSchemaId: Short, valueSchema: StructType, keyStateEncoderSpec: Option[KeyStateEncoderSpec] = None, userKeyEncoderSchema: Option[StructType] = None) extends Product with Serializable
80.  class StateStoreColumnFamilyMismatch extends SparkUnsupportedOperationException
81.  class StateStoreConf extends Serializable

    A class that contains configuration parameters for StateStores.

82.  class StateStoreCoordinatorRef extends AnyRef

    Reference to a StateStoreCoordinator that can be used to coordinate instances of StateStores across all the executors, and get their locations for job scheduling.

83.  trait StateStoreCustomMetric extends AnyRef

    Name and description of custom implementation-specific metrics that a state store may wish to expose.

    Name and description of custom implementation-specific metrics that a state store may wish to expose. Also provides SQLMetric instance to show the metric in UI and accumulate it at the query level.

84.  case class StateStoreCustomSizeMetric(name: String, desc: String) extends StateStoreCustomMetric with Product with Serializable
85.  case class StateStoreCustomSumMetric(name: String, desc: String) extends StateStoreCustomMetric with Product with Serializable
86.  case class StateStoreCustomTimingMetric(name: String, desc: String) extends StateStoreCustomMetric with Product with Serializable
87.  class StateStoreDuplicateStateVariableDefined extends SparkRuntimeException
88.  sealed trait StateStoreEncoding extends AnyRef
89.  case class StateStoreId(checkpointRootLocation: String, operatorId: Long, partitionId: Int, storeName: String = StateStoreId.DEFAULT_STORE_NAME) extends Product with Serializable

    Unique identifier for a bunch of keyed state data.

    Unique identifier for a bunch of keyed state data.

    checkpointRootLocation

    Root directory where all the state data of a query is stored

    operatorId

    Unique id of a stateful operator

    partitionId

    Index of the partition of an operators state data

    storeName

    Optional, name of the store. Each partition can optionally use multiple state stores, but they have to be identified by distinct names.

90.  class StateStoreIncorrectNumOrderingColsForPrefixScan extends SparkUnsupportedOperationException
91.  class StateStoreIncorrectNumOrderingColsForRangeScan extends SparkUnsupportedOperationException
92.  class StateStoreInvalidConfigAfterRestart extends SparkUnsupportedOperationException
93.  class StateStoreInvalidValueSchemaEvolution extends SparkUnsupportedOperationException
94.  class StateStoreInvalidVariableTypeChange extends SparkUnsupportedOperationException
95.  class StateStoreIterator[A] extends Iterator[A] with Closeable

    Represents an iterator that provides additional functionalities for state store use cases.

    Represents an iterator that provides additional functionalities for state store use cases.

    close() is useful for freeing underlying iterator resources when the iterator is no longer needed.

    The caller MUST call close() on the iterator if it was not fully consumed, and it is no longer needed.

96.  class StateStoreKeyRowFormatValidationFailure extends SparkRuntimeException
97.  class StateStoreKeySchemaNotCompatible extends SparkUnsupportedOperationException
98.  trait StateStoreMetadata extends AnyRef

    Metadata for a state store instance.

99.  case class StateStoreMetadataV1(storeName: String, numColsPrefixKey: Int, numPartitions: Int) extends StateStoreMetadata with Product with Serializable
100.  case class StateStoreMetadataV2(storeName: String, numColsPrefixKey: Int, numPartitions: Int, stateSchemaFilePaths: List[String]) extends StateStoreMetadata with Serializable with Product
101.  case class StateStoreMetrics(numKeys: Long, memoryUsedBytes: Long, customMetrics: Map[StateStoreCustomMetric, Long]) extends Product with Serializable

     Metrics reported by a state store

     Metrics reported by a state store

     numKeys

     Number of keys in the state store

     memoryUsedBytes

     Memory used by the state store

     customMetrics

     Custom implementation-specific metrics The metrics reported through this must have the same name as those reported by StateStoreProvider.customMetrics.

102.  class StateStoreMultipleColumnFamiliesNotSupportedException extends SparkUnsupportedOperationException
103.  class StateStoreNullTypeOrderingColsNotSupported extends SparkUnsupportedOperationException
104.  class StateStoreOperationOutOfOrder extends SparkRuntimeException
105.  implicit class StateStoreOps[T] extends AnyRef
106.  trait StateStoreProvider extends AnyRef

     Trait representing a provider that provide StateStore instances representing versions of state data.

     Trait representing a provider that provide StateStore instances representing versions of state data.

     The life cycle of a provider and its provide stores are as follows.

     - A StateStoreProvider is created in a executor for each unique StateStoreId when the first batch of a streaming query is executed on the executor. All subsequent batches reuse this provider instance until the query is stopped.

     - Every batch of streaming data request a specific version of the state data by invoking getStore(version) which returns an instance of StateStore through which the required version of the data can be accessed. It is the responsible of the provider to populate this store with context information like the schema of keys and values, etc.

     If the checkpoint format version 2 is used, an additional argument checkpointID may be provided as part of getStore(version, checkpointID). The provider needs to guarantee that the loaded version is of this unique ID. It needs to load the version for this specific ID from the checkpoint if needed.

     - After the streaming query is stopped, the created provider instances are lazily disposed off.

107.  class StateStoreProviderDoesNotSupportFineGrainedReplay extends SparkUnsupportedOperationException
108.  case class StateStoreProviderId(storeId: StateStoreId, queryRunId: UUID) extends Product with Serializable

     Unique identifier for a provider, used to identify when providers can be reused.

     Unique identifier for a provider, used to identify when providers can be reused. Note that queryRunId is used uniquely identify a provider, so that the same provider instance is not reused across query restarts.

109.  class StateStoreRDD[T, U] extends BaseStateStoreRDD[T, U]

     An RDD that allows computations to be executed against StateStores.

     An RDD that allows computations to be executed against StateStores. It uses the StateStoreCoordinator to get the locations of loaded state stores and use that as the preferred locations.

110.  class StateStoreRemovingColumnFamiliesNotSupportedException extends SparkUnsupportedOperationException
111.  class StateStoreSnapshotFileNotFound extends SparkRuntimeException
112.  class StateStoreSnapshotPartitionNotFound extends SparkRuntimeException
113.  class StateStoreStateSchemaFilesThresholdExceeded extends SparkUnsupportedOperationException
114.  class StateStoreUnsupportedOperationException extends SparkUnsupportedOperationException
115.  class StateStoreUnsupportedOperationOnMissingColumnFamily extends SparkUnsupportedOperationException
116.  class StateStoreValueRowFormatValidationFailure extends SparkRuntimeException
117.  class StateStoreValueSchemaEvolutionThresholdExceeded extends SparkUnsupportedOperationException
118.  class StateStoreValueSchemaNotCompatible extends SparkUnsupportedOperationException
119.  class StateStoreVariableSizeOrderingColsNotSupported extends SparkUnsupportedOperationException
120.  class StatefulProcessorCannotAssignTTLInTimeMode extends SparkUnsupportedOperationException
121.  class StatefulProcessorCannotPerformOperationWithInvalidHandleState extends SparkUnsupportedOperationException
122.  class StatefulProcessorCannotPerformOperationWithInvalidTimeMode extends SparkUnsupportedOperationException
123.  class StatefulProcessorTTLMustBePositive extends SparkUnsupportedOperationException
124.  sealed trait StreamingAggregationStateManager extends Serializable

     Base trait for state manager purposed to be used from streaming aggregations.

125.  abstract class StreamingAggregationStateManagerBaseImpl extends StreamingAggregationStateManager
126.  class StreamingAggregationStateManagerImplV1 extends StreamingAggregationStateManagerBaseImpl

     The implementation of StreamingAggregationStateManager for state version 1.

     The implementation of StreamingAggregationStateManager for state version 1. In state version 1, the schema of key and value in state are follow:

     - key: Same as key expressions. - value: Same as input row attributes. The schema of value contains key expressions as well.

127.  class StreamingAggregationStateManagerImplV2 extends StreamingAggregationStateManagerBaseImpl

     The implementation of StreamingAggregationStateManager for state version 2.

     The implementation of StreamingAggregationStateManager for state version 2. In state version 2, the schema of key and value in state are follow:

     - key: Same as key expressions. - value: The diff between input row attributes and key expressions.

     The schema of value is changed to optimize the memory/space usage in state, via removing duplicated columns in key-value pair. Hence key columns are excluded from the schema of value.

128.  class StreamingSessionWindowHelper extends AnyRef
129.  sealed trait StreamingSessionWindowStateManager extends Serializable
130.  class StreamingSessionWindowStateManagerImplV1 extends StreamingSessionWindowStateManager with Logging
131.  trait SupportsFineGrainedReplay extends AnyRef

     This is an optional trait to be implemented by StateStoreProviders that can read the change of state store over batches.

     This is an optional trait to be implemented by StateStoreProviders that can read the change of state store over batches. This is used by State Data Source with additional options like snapshotStartBatchId or readChangeFeed.

132.  class SymmetricHashJoinStateManager extends Logging

     Helper class to manage state required by a single side of org.apache.spark.sql.execution.streaming.StreamingSymmetricHashJoinExec.

     Helper class to manage state required by a single side of org.apache.spark.sql.execution.streaming.StreamingSymmetricHashJoinExec. The interface of this class is basically that of a multi-map: - Get: Returns an iterator of multiple values for given key - Append: Append a new value to the given key - Remove Data by predicate: Drop any state using a predicate condition on keys or values

133.  class TWSSchemaMustBeNullable extends SparkUnsupportedOperationException
134.  class TestStateSchemaProvider extends StateSchemaProvider
135.  class UnsafeRowDataEncoder extends RocksDBDataEncoder
136.  class UnsafeRowPair extends AnyRef

     Mutable, and reusable class for representing a pair of UnsafeRows.

137.  class WrappedReadStateStore extends ReadStateStore

     Wraps the instance of StateStore to make the instance read-only.