org.apache.spark.sql.catalyst.plans.physical
KeyGroupedShuffleSpec
Companion object KeyGroupedShuffleSpec
case class KeyGroupedShuffleSpec(partitioning: KeyGroupedPartitioning, distribution: ClusteredDistribution, joinKeyPositions: Option[Seq[Int]] = None) extends ShuffleSpec with Product with Serializable
ShuffleSpec created by KeyGroupedPartitioning.
- partitioning
key grouped partitioning
- distribution
distribution
- joinKeyPositions
position of join keys among cluster keys. This is set if joining on a subset of cluster keys is allowed.
- Alphabetic
- By Inheritance
- KeyGroupedShuffleSpec
- Serializable
- Product
- Equals
- ShuffleSpec
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new KeyGroupedShuffleSpec(partitioning: KeyGroupedPartitioning, distribution: ClusteredDistribution, joinKeyPositions: Option[Seq[Int]] = None)
- partitioning
key grouped partitioning
- distribution
distribution
- joinKeyPositions
position of join keys among cluster keys. This is set if joining on a subset of cluster keys is allowed.
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- def areKeysCompatible(other: KeyGroupedShuffleSpec): Boolean
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def canCreatePartitioning: Boolean
Whether this shuffle spec can be used to create partitionings for the other children.
Whether this shuffle spec can be used to create partitionings for the other children.
- Definition Classes
- KeyGroupedShuffleSpec → ShuffleSpec
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- def createPartitioning(clustering: Seq[Expression]): Partitioning
Creates a partitioning that can be used to re-partition the other side with the given clustering expressions.
Creates a partitioning that can be used to re-partition the other side with the given clustering expressions.
This will only be called when:
- isCompatibleWith returns false on the side where the
clusteringis from.
- Definition Classes
- KeyGroupedShuffleSpec → ShuffleSpec
- isCompatibleWith returns false on the side where the
- val distribution: ClusteredDistribution
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def isCompatibleWith(other: ShuffleSpec): Boolean
Returns true iff this spec is compatible with the provided shuffle spec.
Returns true iff this spec is compatible with the provided shuffle spec.
A true return value means that the data partitioning from this spec can be seen as co-partitioned with the
other, and therefore no shuffle is required when joining the two sides.Note that Spark assumes this to be reflexive, symmetric and transitive.
- Definition Classes
- KeyGroupedShuffleSpec → ShuffleSpec
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- val joinKeyPositions: Option[Seq[Int]]
- lazy val keyPositions: Seq[BitSet]
A sequence where each element is a set of positions of the partition expression to the cluster keys.
A sequence where each element is a set of positions of the partition expression to the cluster keys. For instance, if cluster keys are [a, b, b] and partition expressions are [bucket(4, a), years(b)], the result will be [(0), (1, 2)].
Note that we only allow each partition expression to contain a single partition key. Therefore the mapping here is very similar to that from
HashShuffleSpec. - final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- def numPartitions: Int
Returns the number of partitions of this shuffle spec
Returns the number of partitions of this shuffle spec
- Definition Classes
- KeyGroupedShuffleSpec → ShuffleSpec
- val partitioning: KeyGroupedPartitioning
- def productElementNames: Iterator[String]
- Definition Classes
- Product
- def reducers(other: KeyGroupedShuffleSpec): Option[Seq[Option[Reducer[_, _]]]]
Return a set of Reducer for the partition expressions of this shuffle spec, on the partition expressions of another shuffle spec.
Return a set of Reducer for the partition expressions of this shuffle spec, on the partition expressions of another shuffle spec.
A Reducer exists for a partition expression function of this shuffle spec if it is 'reducible' on the corresponding partition expression function of the other shuffle spec.
If a value is returned, there must be one Reducer per partition expression. A None value in the set indicates that the particular partition expression is not reducible on the corresponding expression on the other shuffle spec.
Returning none also indicates that none of the partition expressions can be reduced on the corresponding expression on the other shuffle spec.
- other
other key-grouped shuffle spec
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)