object SchemaUtil
Linear Supertypes
Ordering
- Alphabetic
- By Inheritance
Inherited
- SchemaUtil
- AnyRef
- Any
- Hide All
- Show All
Visibility
- Public
- Protected
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def checkVariableType(stateVariableInfoOpt: Option[TransformWithStateVariableInfo], varType: StateVariableType): Boolean
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def getCompositeKeySchema(schema: StructType, stateSourceOptions: StateSourceOptions): StructType
Given key-value schema generated from
generateSchemaForStateVar(), returns the compositeKey schema that key is stored in the state store - def getSchemaAsDataType(schema: StructType, fieldName: String): DataType
- def getSourceSchema(sourceOptions: StateSourceOptions, keySchema: StructType, valueSchema: StructType, transformWithStateVariableInfoOpt: Option[TransformWithStateVariableInfo], stateStoreColFamilySchemaOpt: Option[StateStoreColFamilySchema]): StructType
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- def isValidSchema(sourceOptions: StateSourceOptions, schema: StructType, transformWithStateVariableInfoOpt: Option[TransformWithStateVariableInfo]): Boolean
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- def processStateEntries(stateVarType: StateVariableType, stateVarName: String, store: ReadStateStore, compositeKeySchema: StructType, partitionId: Int, stateSourceOptions: StateSourceOptions): Iterator[InternalRow]
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- def unifyMapStateRowPair(stateRows: Iterator[UnsafeRowPair], compositeKeySchema: StructType, partitionId: Int, stateSourceOptions: StateSourceOptions): Iterator[InternalRow]
For map state variables, state rows are stored as composite key.
For map state variables, state rows are stored as composite key. To return grouping key -> Map{user key -> value} as one state reader row to the users, we need to perform grouping on state rows by their grouping key, and construct a map for that grouping key.
We traverse the iterator returned from state store, and will only return a row for
next()only if the grouping key in the next row from state store is different (or there are no more rows)Note that all state rows with the same grouping key are co-located so they will appear consecutively during the iterator traversal.
- def unifyStateRowPair(pair: (UnsafeRow, UnsafeRow), partition: Int): InternalRow
- def unifyStateRowPairWithMultipleValues(pair: (UnsafeRow, GenericArrayData), partition: Int): InternalRow
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)