Packages

abstract class SymmetricHashJoinStateManager extends Logging

Helper class to manage state required by a single side of org.apache.spark.sql.execution.streaming.StreamingSymmetricHashJoinExec. The interface of this class is basically that of a multi-map: - Get: Returns an iterator of multiple values for given key - Append: Append a new value to the given key - Remove Data by predicate: Drop any state using a predicate condition on keys or values

Linear Supertypes
Logging, AnyRef, Any
Known Subclasses
SymmetricHashJoinStateManagerV1, SymmetricHashJoinStateManagerV2
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SymmetricHashJoinStateManager
  2. Logging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new SymmetricHashJoinStateManager(joinSide: JoinSide, inputValueAttributes: Seq[Attribute], joinKeys: Seq[Expression], stateInfo: Option[StatefulOperatorStateInfo], storeConf: StateStoreConf, hadoopConf: Configuration, partitionId: Int, keyToNumValuesStateStoreCkptId: Option[String], keyWithIndexToValueStateStoreCkptId: Option[String], stateFormatVersion: Int, skippedNullValueCount: Option[SQLMetric] = None, useStateStoreCoordinator: Boolean = true, snapshotOptions: Option[SnapshotOptions] = None, joinStoreGenerator: JoinStateManagerStoreGenerator)

    joinSide

    Defines the join side

    inputValueAttributes

    Attributes of the input row which will be stored as value

    joinKeys

    Expressions to generate rows that will be used to key the value rows

    stateInfo

    Information about how to retrieve the correct version of state

    storeConf

    Configuration for the state store.

    hadoopConf

    Hadoop configuration for reading state data from storage

    partitionId

    A partition ID of source RDD.

    stateFormatVersion

    The version of format for state.

    skippedNullValueCount

    The instance of SQLMetric tracking the number of skipped null values.

    useStateStoreCoordinator

    Whether to use a state store coordinator to maintain the state store providers being used in this class. If true, Spark will take care of management for state store providers, e.g. running maintenance task for these providers.

    snapshotOptions

    Options controlling snapshot-based state replay for the state data source reader.

    joinStoreGenerator

    The generator to create state store instances, re-using the same instance when the join implementation uses virtual column families for join version 3. Internally, the key -> multiple values is stored in two StateStores. - Store 1 (KeyToNumValuesStore) maintains mapping between key -> number of values - Store 2 (KeyWithIndexToValueStore) maintains mapping; the mapping depends on the state format version:

    • version 1: [(key, index) -> value]
    • version 2: [(key, index) -> (value, matched)] - Put: update count in KeyToNumValuesStore, insert new (key, count) -> value in KeyWithIndexToValueStore - Get: read count from KeyToNumValuesStore, read each of the n values in KeyWithIndexToValueStore - Remove state by predicate on keys: scan all keys in KeyToNumValuesStore to find keys that do match the predicate, delete from key from KeyToNumValuesStore, delete values in KeyWithIndexToValueStore - Remove state by condition on values: scan all elements in KeyWithIndexToValueStore to find values that match the predicate, delete corresponding (key, indexToDelete) from KeyWithIndexToValueStore by overwriting with the value of (key, maxIndex), and removing [(key, maxIndex), decrement corresponding num values in KeyToNumValuesStore

Type Members

  1. class KeyToNumValuesStore extends StateStoreHandler

    A wrapper around a StateStore that stores [key -> number of values].

    A wrapper around a StateStore that stores [key -> number of values].

    Attributes
    protected
  2. class KeyWithIndexToValueStore extends StateStoreHandler

    A wrapper around a StateStore that stores the mapping; the mapping depends on the state format version - please refer implementations of KeyWithIndexToValueRowConverter.

    A wrapper around a StateStore that stores the mapping; the mapping depends on the state format version - please refer implementations of KeyWithIndexToValueRowConverter.

    Attributes
    protected
  3. implicit class LogStringContext extends AnyRef
    Definition Classes
    Logging
  4. abstract class StateStoreHandler extends Logging

    Helper trait for invoking common functionalities of a state store.

    Helper trait for invoking common functionalities of a state store.

    Attributes
    protected

Abstract Value Members

  1. abstract def abortIfNeeded(): Unit

    Abort any changes to the state stores if needed

  2. abstract def commit(): Unit

    Commit all the changes to all the state stores

  3. abstract def getLatestCheckpointInfo(): JoinerStateStoreCkptInfo

    Get state store checkpoint information of the two state stores for this joiner, after they finished data processing.

    Get state store checkpoint information of the two state stores for this joiner, after they finished data processing.

    For SymmetricHashJoinStateManagerV1, this returns the information of the two stores used for this joiner.

    For SymmetricHashJoinStateManagerV2, this returns the information of the single store used for the entire joiner operator. Both fields of JoinerStateStoreCkptInfo will be identical.

  4. abstract def metrics: StateStoreMetrics

    Get the combined metrics of all the state stores

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def MDC(key: LogKey, value: Any): MDC
    Attributes
    protected
    Definition Classes
    Logging
  5. def append(key: UnsafeRow, value: UnsafeRow, matched: Boolean): Unit

    Append a new value to the key

  6. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  10. def get(key: UnsafeRow): Iterator[UnsafeRow]

    Get all the values of a key

  11. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  12. def getInternalRowOfKeyWithIndex(currentKey: UnsafeRow): InternalRow

    Projects the key of unsafe row to internal row for printable log message.

  13. def getJoinedRows(key: UnsafeRow, generateJoinedRow: (InternalRow) => JoinedRow, predicate: (JoinedRow) => Boolean, excludeRowsAlreadyMatched: Boolean = false): Iterator[JoinedRow]

    Get all the matched values for given join condition, with marking matched.

    Get all the matched values for given join condition, with marking matched. This method is designed to mark joined rows properly without exposing internal index of row.

    excludeRowsAlreadyMatched

    Do not join with rows already matched previously. This is used for right side of left semi join in StreamingSymmetricHashJoinExec only.

  14. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  15. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  16. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  17. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  18. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  19. def iterator: Iterator[KeyToValuePair]

    Perform a full scan to provide all available data.

    Perform a full scan to provide all available data.

    This produces an iterator over the (key, value, match) tuples. Callers are expected to consume fully to clean up underlying iterators correctly.

  20. val keyAttributes: Seq[AttributeReference]
    Attributes
    protected
  21. val keySchema: StructType
    Attributes
    protected
  22. val keyToNumValues: KeyToNumValuesStore
    Attributes
    protected
  23. val keyWithIndexToValue: KeyWithIndexToValueStore
    Attributes
    protected
  24. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  25. def logBasedOnLevel(level: Level)(f: => MessageWithContext): Unit
    Attributes
    protected
    Definition Classes
    Logging
  26. def logDebug(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  27. def logDebug(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  28. def logDebug(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  29. def logDebug(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  30. def logError(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  31. def logError(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  32. def logError(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  33. def logError(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  34. def logInfo(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  35. def logInfo(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  36. def logInfo(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  37. def logInfo(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  38. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  39. def logTrace(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  40. def logTrace(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  41. def logTrace(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  42. def logTrace(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  43. def logWarning(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  44. def logWarning(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  45. def logWarning(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  46. def logWarning(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  47. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  48. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  49. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  50. def removeByKeyCondition(removalCondition: (UnsafeRow) => Boolean): Iterator[KeyToValuePair]

    Remove using a predicate on keys.

    Remove using a predicate on keys.

    This produces an iterator over the (key, value, matched) tuples satisfying condition(key), where the underlying store is updated as a side-effect of producing next.

    This implies the iterator must be consumed fully without any other operations on this manager or the underlying store being interleaved.

  51. def removeByValueCondition(removalCondition: (UnsafeRow) => Boolean): Iterator[KeyToValuePair]

    Remove using a predicate on values.

    Remove using a predicate on values.

    At a high level, this produces an iterator over the (key, value, matched) tuples such that value satisfies the predicate, where producing an element removes the value from the state store and producing all elements with a given key updates it accordingly.

    This implies the iterator must be consumed fully without any other operations on this manager or the underlying store being interleaved.

  52. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  53. def toString(): String
    Definition Classes
    AnyRef → Any
  54. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  55. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  56. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  57. def withLogContext(context: Map[String, String])(body: => Unit): Unit
    Attributes
    protected
    Definition Classes
    Logging

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped