Packages

class AvroStateEncoder extends RocksDBDataEncoder with Logging

Encoder that uses Avro for serializing state store data with schema evolution support. The encoded format varies depending on the key type and whether it's a key or value:

For prefix and range scan keys:

--prefix---

--schemaId (2 bytes)--

--remainingKeyBytes (avro-encoded)--

where: - prefix: Variable length prefix for scan operations - schemaId: 2 byte short integer identifying the schema version - remainingKeyBytes: Avro-encoded remaining key data

For no-prefix keys and values:

--schemaId (2 bytes)--

--avroEncodedBytes--

where: - schemaId: 2 byte short integer identifying the schema version - avroEncodedBytes: Variable length Avro-encoded data

The schema ID allows the state store to identify which schema version was used to encode the data, enabling proper decoding even when schemas have evolved over time.

Linear Supertypes
Logging, RocksDBDataEncoder, DataEncoder, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. AvroStateEncoder
  2. Logging
  3. RocksDBDataEncoder
  4. DataEncoder
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new AvroStateEncoder(keyStateEncoderSpec: KeyStateEncoderSpec, valueSchema: StructType, stateSchemaProvider: Option[StateSchemaProvider], columnFamilyName: String)

    keyStateEncoderSpec

    Specification for how to encode keys (prefix/range scan)

    valueSchema

    Schema for the values to be encoded

    stateSchemaProvider

    Optional state schema provider

    columnFamilyName

    Column family name to be used

Type Members

  1. implicit class LogStringContext extends AnyRef
    Definition Classes
    Logging

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  6. def decodeFromAvroToUnsafeRow(valueBytes: Array[Byte], avroDeserializer: AvroDeserializer, writerSchema: Schema, readerSchema: Schema, valueProj: UnsafeProjection): UnsafeRow

    This method takes a byte array written using Avro encoding, and deserializes to an UnsafeRow using the Avro deserializer

    This method takes a byte array written using Avro encoding, and deserializes to an UnsafeRow using the Avro deserializer

    valueBytes

    The raw bytes containing Avro-encoded data

    avroDeserializer

    Custom deserializer to convert Avro records to InternalRows

    writerSchema

    The Avro schema used when writing the data

    readerSchema

    The Avro schema to use for reading (may be different from writer schema)

    valueProj

    Projection to convert InternalRow to UnsafeRow

    returns

    The deserialized UnsafeRow, or null if input bytes are null

  7. def decodeFromAvroToUnsafeRow(valueBytes: Array[Byte], avroDeserializer: AvroDeserializer, valueAvroType: Schema, valueProj: UnsafeProjection): UnsafeRow

    This method takes a byte array written using Avro encoding, and deserializes to an UnsafeRow using the Avro deserializer

  8. def decodeKey(rowBytes: Array[Byte]): UnsafeRow

    Decodes a complete key from its serialized byte form.

    Decodes a complete key from its serialized byte form.

    For NoPrefixKeyStateEncoder: Decodes the entire key For PrefixKeyScanStateEncoder: Decodes only the prefix portion

    returns

    UnsafeRow containing the decoded key columns

    Definition Classes
    AvroStateEncoder → DataEncoder
    Exceptions thrown

    UnsupportedOperationException for unsupported encoder types

  9. def decodePrefixKeyForRangeScan(rowBytes: Array[Byte]): UnsafeRow

    Decodes an Avro-encoded byte array back into an UnsafeRow for range scan operations.

    Decodes an Avro-encoded byte array back into an UnsafeRow for range scan operations.

    This method reverses the encoding process performed by encodePrefixKeyForRangeScan: - Reads the marker byte to determine null status or sign - Reconstructs the original values from big-endian format - Handles special cases for floating point numbers by reversing bit manipulations

    The decoding process preserves the original data types and values, including: - Null values marked by nullValMarker - Sign information for numeric types - Proper restoration of negative floating point values

    returns

    UnsafeRow containing the decoded data

    Definition Classes
    AvroStateEncoder → DataEncoder
    Exceptions thrown

    UnsupportedOperationException if a field's data type is not supported for range scan decoding

  10. def decodeRemainingKey(rowBytes: Array[Byte]): UnsafeRow

    Decodes the remaining portion of a split key from its serialized form.

    Decodes the remaining portion of a split key from its serialized form.

    For PrefixKeyScanStateEncoder: Decodes columns after the prefix For RangeKeyScanStateEncoder: Decodes non-ordering columns

    returns

    UnsafeRow containing the decoded remaining key columns

    Definition Classes
    AvroStateEncoder → DataEncoder
    Exceptions thrown

    UnsupportedOperationException if called on an encoder that doesn't support split keys

  11. def decodeStateSchemaIdRow(bytes: Array[Byte]): StateSchemaIdRow
    Definition Classes
    RocksDBDataEncoder
  12. def decodeToUnsafeRow(bytes: Array[Byte], reusedRow: UnsafeRow): UnsafeRow
    Definition Classes
    RocksDBDataEncoder
  13. def decodeToUnsafeRow(bytes: Array[Byte], numFields: Int): UnsafeRow
    Definition Classes
    RocksDBDataEncoder
  14. def decodeValue(rowBytes: Array[Byte]): UnsafeRow

    Decodes a value from its serialized byte form.

    Decodes a value from its serialized byte form.

    returns

    UnsafeRow containing the decoded value columns

    Definition Classes
    AvroStateEncoder → DataEncoder
  15. val doubleFlipBitMask: Long
    Definition Classes
    RocksDBDataEncoder
  16. val doubleSignBitMask: Long
    Definition Classes
    RocksDBDataEncoder
  17. def encodeKey(row: UnsafeRow): Array[Byte]

    Encodes a complete key row into bytes.

    Encodes a complete key row into bytes. Used as the primary key for state lookups.

    row

    An UnsafeRow containing all key columns as defined in the keySchema

    returns

    Serialized byte array representation of the key

    Definition Classes
    AvroStateEncoder → DataEncoder
  18. def encodePrefixKeyForRangeScan(row: UnsafeRow): Array[Byte]

    Encodes an UnsafeRow into an Avro-compatible byte array format for range scan operations.

    Encodes an UnsafeRow into an Avro-compatible byte array format for range scan operations.

    This method transforms row data into a binary format that preserves ordering when used in range scans. For each field in the row: - A marker byte is written to indicate null status or sign (for numeric types) - The value is written in big-endian format

    Special handling is implemented for: - Null values: marked with nullValMarker followed by zero bytes - Negative numbers: marked with negativeValMarker - Floating point numbers: bit manipulation to handle sign and NaN values correctly

    row

    The UnsafeRow to encode

    returns

    Array[Byte] containing the Avro-encoded data that preserves ordering for range scans

    Definition Classes
    AvroStateEncoder → DataEncoder
    Exceptions thrown

    UnsupportedOperationException if a field's data type is not supported for range scan encoding

  19. def encodeRemainingKey(row: UnsafeRow): Array[Byte]

    Encodes the non-prefix portion of a key row.

    Encodes the non-prefix portion of a key row. Used with prefix scan and range scan state lookups where the key is split into prefix and remaining portions.

    For prefix scans: Encodes columns after the prefix columns For range scans: Encodes columns not included in the ordering columns

    row

    An UnsafeRow containing only the remaining key columns

    returns

    Serialized byte array of the remaining key portion

    Definition Classes
    AvroStateEncoder → DataEncoder
    Exceptions thrown

    UnsupportedOperationException if called on an encoder that doesn't support split keys

  20. def encodeUnsafeRow(row: UnsafeRow): Array[Byte]

    Encode the UnsafeRow of N bytes as a N+1 byte array.

    Encode the UnsafeRow of N bytes as a N+1 byte array.

    Definition Classes
    RocksDBDataEncoder
    Note

    This creates a new byte array and memcopies the UnsafeRow to the new array.

  21. def encodeUnsafeRowToAvro(row: UnsafeRow, avroSerializer: AvroSerializer, valueAvroType: Schema, out: ByteArrayOutputStream): Array[Byte]

    This method takes an UnsafeRow, and serializes to a byte array using Avro encoding.

  22. def encodeValue(row: UnsafeRow): Array[Byte]

    Encodes a value row into bytes.

    Encodes a value row into bytes.

    row

    An UnsafeRow containing the value columns as defined in the valueSchema

    returns

    Serialized byte array representation of the value

    Definition Classes
    AvroStateEncoder → DataEncoder
  23. def encodeWithStateSchemaId(schemaIdRow: StateSchemaIdRow): Array[Byte]
    Definition Classes
    RocksDBDataEncoder
  24. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  25. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  26. val floatFlipBitMask: Int
    Definition Classes
    RocksDBDataEncoder
  27. val floatSignBitMask: Int
    Definition Classes
    RocksDBDataEncoder
  28. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  29. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  30. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  31. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  32. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  33. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  34. val keySchema: StructType
    Definition Classes
    RocksDBDataEncoder
  35. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  36. def logDebug(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  37. def logDebug(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  38. def logDebug(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  39. def logDebug(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  40. def logError(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  41. def logError(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  42. def logError(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  43. def logError(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  44. def logInfo(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  45. def logInfo(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  46. def logInfo(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  47. def logInfo(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  48. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  49. def logTrace(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  50. def logTrace(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  51. def logTrace(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  52. def logTrace(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  53. def logWarning(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  54. def logWarning(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  55. def logWarning(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  56. def logWarning(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  57. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  58. val negativeValMarker: Byte
    Definition Classes
    RocksDBDataEncoder
  59. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  60. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  61. val nullValMarker: Byte
    Definition Classes
    RocksDBDataEncoder
  62. val positiveValMarker: Byte
    Definition Classes
    RocksDBDataEncoder
  63. val reusedKeyRow: UnsafeRow
    Definition Classes
    RocksDBDataEncoder
  64. val reusedValueRow: UnsafeRow
    Definition Classes
    RocksDBDataEncoder
  65. def supportsSchemaEvolution: Boolean
    Definition Classes
    AvroStateEncoder → DataEncoder
  66. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  67. def toString(): String
    Definition Classes
    AnyRef → Any
  68. def unsupportedOperationForKeyStateEncoder(operation: String): UnsupportedOperationException
    Definition Classes
    RocksDBDataEncoder
  69. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  70. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  71. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  72. def withLogContext(context: Map[String, String])(body: => Unit): Unit
    Attributes
    protected
    Definition Classes
    Logging

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from Logging

Inherited from RocksDBDataEncoder

Inherited from DataEncoder

Inherited from AnyRef

Inherited from Any

Ungrouped