Packages

c

org.apache.spark.sql.execution.streaming

DriverStatefulProcessorHandleImpl

class DriverStatefulProcessorHandleImpl extends StatefulProcessorHandleImplBase

This DriverStatefulProcessorHandleImpl is used within TransformWithExec on the driver side to collect the columnFamilySchemas before any processing is actually done. We need this class because we can only collect the schemas after the StatefulProcessor is initialized.

Linear Supertypes
StatefulProcessorHandleImplBase, StatefulProcessorHandle, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DriverStatefulProcessorHandleImpl
  2. StatefulProcessorHandleImplBase
  3. StatefulProcessorHandle
  4. Serializable
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new DriverStatefulProcessorHandleImpl(timeMode: TimeMode, keyExprEnc: ExpressionEncoder[Any])

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def addTTLSchemas(columnFamilySchemas: Map[String, StateStoreColFamilySchema], stateVariableInfo: TransformWithStateVariableInfo, stateName: String, keySchema: StructType): Unit

    Updates the column family schemas map to handle TTL column families.

  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  7. var currState: StatefulProcessorHandleState
    Attributes
    protected
    Definition Classes
    StatefulProcessorHandleImplBase
  8. def deleteIfExists(stateName: String): Unit
    Definition Classes
    DriverStatefulProcessorHandleImpl → StatefulProcessorHandle
  9. def deleteTimer(expiryTimestampMs: Long): Unit
    Definition Classes
    DriverStatefulProcessorHandleImpl → StatefulProcessorHandle
  10. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  11. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  12. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  13. def getColumnFamilySchemas(shouldCheckNullable: Boolean, shouldSetNullable: Boolean): Map[String, StateStoreColFamilySchema]

    This method returns all column family schemas, and checks and enforces nullability if need be.

    This method returns all column family schemas, and checks and enforces nullability if need be. The nullability check and set is only set to true when Avro is enabled.

    shouldCheckNullable

    Whether we need to check the nullability. This is set to true when using Python, as this is the only avenue through which users can set nullability

    shouldSetNullable

    Whether we need to set the fields as nullable. This is set to true when using Scala, as primitive type encoders set the field to non-nullable. Changing fields from non-nullable to nullable does not break anything (and is required for Avro encoding), so we can safely make this change.

    returns

    column family schemas used by this stateful processor.

  14. def getCountIndexSchema(stateName: String, elementKeySchema: StructType): StateStoreColFamilySchema

    Gets the schema for count index column family which maps elementKey -> count.

    Gets the schema for count index column family which maps elementKey -> count. This is used by one-to-many TTL states to track number of entries.

  15. def getHandleState: StatefulProcessorHandleState
    Definition Classes
    StatefulProcessorHandleImplBase
  16. def getListState[T](stateName: String, ttlConfig: TTLConfig)(implicit arg0: Encoder[T]): ListState[T]
    Definition Classes
    DriverStatefulProcessorHandleImpl → StatefulProcessorHandle
  17. def getListState[T](stateName: String, valEncoder: Encoder[T], ttlConfig: TTLConfig): ListState[T]
    Definition Classes
    DriverStatefulProcessorHandleImpl → StatefulProcessorHandle
  18. def getMapState[K, V](stateName: String, ttlConfig: TTLConfig)(implicit arg0: Encoder[K], arg1: Encoder[V]): MapState[K, V]
    Definition Classes
    DriverStatefulProcessorHandleImpl → StatefulProcessorHandle
  19. def getMapState[K, V](stateName: String, userKeyEnc: Encoder[K], valEncoder: Encoder[V], ttlConfig: TTLConfig): MapState[K, V]
    Definition Classes
    DriverStatefulProcessorHandleImpl → StatefulProcessorHandle
  20. def getMinExpiryIndexSchema(stateName: String, elementKeySchema: StructType): StateStoreColFamilySchema

    Gets the schema for min expiry index column family which maps elementKey -> minExpirationMs.

    Gets the schema for min expiry index column family which maps elementKey -> minExpirationMs. This is used by one-to-many TTL states.

  21. def getQueryInfo(): QueryInfo

    Function to return queryInfo for currently running task

    Function to return queryInfo for currently running task

    Definition Classes
    DriverStatefulProcessorHandleImpl → StatefulProcessorHandle
  22. def getStateVariableInfos: Map[String, TransformWithStateVariableInfo]
  23. def getTTLIndexSchema(stateName: String, elementKeySchema: StructType): StateStoreColFamilySchema

    Gets the schema for TTL index column family which maps (expirationMs, elementKey) -> EMPTY_ROW.

    Gets the schema for TTL index column family which maps (expirationMs, elementKey) -> EMPTY_ROW. This is used by both one-to-one and one-to-many TTL states.

  24. def getValueState[T](stateName: String, ttlConfig: TTLConfig)(implicit arg0: Encoder[T]): ValueState[T]
    Definition Classes
    DriverStatefulProcessorHandleImpl → StatefulProcessorHandle
  25. def getValueState[T](stateName: String, valEncoder: Encoder[T], ttlConfig: TTLConfig): ValueState[T]
    Definition Classes
    DriverStatefulProcessorHandleImpl → StatefulProcessorHandle
  26. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  27. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  28. def listTimers(): Iterator[Long]
    Definition Classes
    DriverStatefulProcessorHandleImpl → StatefulProcessorHandle
  29. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  30. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  31. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  32. def registerTimer(expiryTimestampMs: Long): Unit

    Methods that are only included to satisfy the interface.

    Methods that are only included to satisfy the interface. These methods will fail if called from the driver side, as the handle will be in the PRE_INIT phase, and all these timer operations need to be called from the INITIALIZED phase.

    Definition Classes
    DriverStatefulProcessorHandleImpl → StatefulProcessorHandle
  33. def setHandleState(newState: StatefulProcessorHandleState): Unit
    Definition Classes
    StatefulProcessorHandleImplBase
  34. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  35. def toString(): String
    Definition Classes
    AnyRef → Any
  36. def verifyStateVarOperations(operationType: String, requiredState: StatefulProcessorHandleState): Unit
    Definition Classes
    StatefulProcessorHandleImplBase
  37. def verifyTimerOperations(operationType: String): Unit
    Definition Classes
    StatefulProcessorHandleImplBase
  38. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  39. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  40. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from StatefulProcessorHandleImplBase

Inherited from StatefulProcessorHandle

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped