Packages

object AnsiTypeCoercion extends TypeCoercionBase

In Spark ANSI mode, the type coercion rules are based on the type precedence lists of the input data types. As per the section "Type precedence list determination" of "ISO/IEC 9075-2:2011 Information technology - Database languages - SQL - Part 2: Foundation (SQL/Foundation)", the type precedence lists of primitive data types are as following: * Byte: Byte, Short, Int, Long, Decimal, Float, Double * Short: Short, Int, Long, Decimal, Float, Double * Int: Int, Long, Decimal, Float, Double * Long: Long, Decimal, Float, Double * Decimal: Float, Double, or any wider Numeric type * Float: Float, Double * Double: Double * String: String * Date: Date, Timestamp * Timestamp: Timestamp * Binary: Binary * Boolean: Boolean * Interval: Interval As for complex data types, Spark will determine the precedent list recursively based on their sub-types and nullability.

With the definition of type precedent list, the general type coercion rules are as following: * Data type S is allowed to be implicitly cast as type T iff T is in the precedence list of S * Comparison is allowed iff the data type precedence list of both sides has at least one common element. When evaluating the comparison, Spark casts both sides as the tightest common data type of their precedent lists. * There should be at least one common data type among all the children's precedence lists for the following operators. The data type of the operator is the tightest common precedent data type. * In * Except * Intersect * Greatest * Least * Union * If * CaseWhen * CreateArray * Array Concat * Sequence * MapConcat * CreateMap * For complex types (struct, array, map), Spark recursively looks into the element type and applies the rules above. Note: this new type coercion system will allow implicit converting String type as other primitive types, in case of breaking too many existing Spark SQL queries. This is a special rule and it is not from the ANSI SQL standard.

Linear Supertypes
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. AnsiTypeCoercion
  2. TypeCoercionBase
  3. TypeCoercionHelper
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Type Members

  1. class AnsiCombinedTypeCoercionRule extends CombinedTypeCoercionRule
  2. class CombinedTypeCoercionRule extends Rule[LogicalPlan] with TypeCoercionRule

    Type coercion rule that combines multiple type coercion rules and applies them in a single tree traversal.

    Type coercion rule that combines multiple type coercion rules and applies them in a single tree traversal.

    Definition Classes
    TypeCoercionBase

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def canCast(from: DataType, to: DataType): Boolean

    Whether casting from as to is valid.

    Whether casting from as to is valid.

    Definition Classes
    AnsiTypeCoercionTypeCoercionHelper
  6. def castIfNotSameType(expr: Expression, dt: DataType): Expression
    Attributes
    protected
    Definition Classes
    TypeCoercionHelper
  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  10. val findTightestCommonType: (DataType, DataType) => Option[DataType]

    Find the tightest common type of two types that might be used in a binary expression.

    Find the tightest common type of two types that might be used in a binary expression. This handles all numeric types except fixed-precision decimals interacting with each other or with primitive types, because in that case the precision and scale of the result depends on the operation. Those rules are implemented in DecimalPrecision.

    Definition Classes
    AnsiTypeCoercionTypeCoercionHelper
  11. def findTypeForComplex(t1: DataType, t2: DataType, findTypeFunc: (DataType, DataType) => Option[DataType]): Option[DataType]
    Attributes
    protected
    Definition Classes
    TypeCoercionHelper
  12. def findWiderCommonType(types: Seq[DataType]): Option[DataType]

    Looking for a widened data type of a given sequence of data types with some acceptable loss of precision.

    Looking for a widened data type of a given sequence of data types with some acceptable loss of precision. E.g. there is no common type for double and decimal because double's range is larger than decimal, and yet decimal is more precise than double, but in union we would cast the decimal into double.

    Definition Classes
    AnsiTypeCoercionTypeCoercionHelper
  13. def findWiderDateTimeType(d1: DatetimeType, d2: DatetimeType): DatetimeType
    Attributes
    protected
    Definition Classes
    TypeCoercionHelper
  14. def findWiderTypeForDecimal(dt1: DataType, dt2: DataType): Option[DataType]

    Finds a wider type when one or both types are decimals.

    Finds a wider type when one or both types are decimals. If the wider decimal type exceeds system limitation, this rule will truncate the decimal type. If a decimal and other fractional types are compared, returns a double type.

    Attributes
    protected
    Definition Classes
    TypeCoercionHelper
  15. def findWiderTypeForTwo(t1: DataType, t2: DataType): Option[DataType]

    Looking for a widened data type of two given data types with some acceptable loss of precision.

    Looking for a widened data type of two given data types with some acceptable loss of precision. E.g. there is no common type for double and decimal because double's range is larger than decimal, and yet decimal is more precise than double, but in union we would cast the decimal into double.

    Definition Classes
    AnsiTypeCoercionTypeCoercionHelper
  16. def findWiderTypeWithoutStringPromotion(types: Seq[DataType]): Option[DataType]
    Definition Classes
    TypeCoercionHelper
  17. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  18. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  19. def haveSameType(types: Seq[DataType]): Boolean

    Check whether the given types are equal ignoring nullable, containsNull and valueContainsNull.

    Check whether the given types are equal ignoring nullable, containsNull and valueContainsNull.

    Definition Classes
    TypeCoercionHelper
  20. def implicitCast(e: Expression, expectedType: AbstractDataType): Option[Expression]

    Given an expected data type, try to cast the expression and return the cast expression.

    Given an expected data type, try to cast the expression and return the cast expression.

    If the expression already fits the input type, we simply return the expression itself. If the expression has an incompatible type that cannot be implicitly cast, return None.

    Definition Classes
    AnsiTypeCoercionTypeCoercionHelper
  21. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  22. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  23. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  24. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  25. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  26. def toString(): String
    Definition Classes
    AnyRef → Any
  27. def typeCoercionRules: List[Rule[LogicalPlan]]

    A collection of Rule that can be used to coerce differing types that participate in operations into compatible ones.

    A collection of Rule that can be used to coerce differing types that participate in operations into compatible ones.

    Definition Classes
    AnsiTypeCoercionTypeCoercionHelper
  28. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  29. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  30. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  31. object DateTimeOperations extends Rule[LogicalPlan] with TypeCoercionRule
  32. object GetDateFieldOperations extends Rule[LogicalPlan] with TypeCoercionRule

    When getting a date field from a Timestamp column, cast the column as date type.

    When getting a date field from a Timestamp column, cast the column as date type.

    This is Spark's hack to make the implementation simple. In the default type coercion rules, the implicit cast rule does the work. However, The ANSI implicit cast rule doesn't allow converting Timestamp type as Date type, so we need to have this additional rule to make sure the date field extraction from Timestamp columns works.

  33. object PromoteStrings extends Rule[LogicalPlan] with TypeCoercionRule
  34. object CaseWhenCoercion extends Rule[LogicalPlan] with TypeCoercionRule

    Coerces the type of different branches of a CASE WHEN statement to a common type.

    Coerces the type of different branches of a CASE WHEN statement to a common type.

    Definition Classes
    TypeCoercionBase
  35. object ConcatCoercion extends Rule[LogicalPlan] with TypeCoercionRule

    Coerces the types of Concat children to expected ones.

    Coerces the types of Concat children to expected ones.

    If spark.sql.function.concatBinaryAsString is false and all children types are binary, the expected types are binary. Otherwise, the expected ones are strings.

    Definition Classes
    TypeCoercionBase
  36. object Division extends Rule[LogicalPlan] with TypeCoercionRule

    Hive only performs integral division with the DIV operator.

    Hive only performs integral division with the DIV operator. The arguments to / are always converted to fractional types.

    Definition Classes
    TypeCoercionBase
  37. object EltCoercion extends Rule[LogicalPlan] with TypeCoercionRule

    Coerces the types of Elt children to expected ones.

    Coerces the types of Elt children to expected ones.

    If spark.sql.function.eltOutputAsString is false and all children types are binary, the expected types are binary. Otherwise, the expected ones are strings.

    Definition Classes
    TypeCoercionBase
  38. object FunctionArgumentConversion extends Rule[LogicalPlan] with TypeCoercionRule

    This ensure that the types for various functions are as expected.

    This ensure that the types for various functions are as expected.

    Definition Classes
    TypeCoercionBase
  39. object IfCoercion extends Rule[LogicalPlan] with TypeCoercionRule

    Coerces the type of different branches of If statement to a common type.

    Coerces the type of different branches of If statement to a common type.

    Definition Classes
    TypeCoercionBase
  40. object ImplicitTypeCasts extends Rule[LogicalPlan] with TypeCoercionRule

    Casts types according to the expected input types for Expressions.

    Casts types according to the expected input types for Expressions.

    Definition Classes
    TypeCoercionBase
  41. object InConversion extends Rule[LogicalPlan] with TypeCoercionRule

    Handles type coercion for both IN expression with subquery and IN expressions without subquery.

    Handles type coercion for both IN expression with subquery and IN expressions without subquery. 1. In the first case, find the common type by comparing the left hand side (LHS) expression types against corresponding right hand side (RHS) expression derived from the subquery expression's plan output. Inject appropriate casts in the LHS and RHS side of IN expression.

    2. In the second case, convert the value and in list expressions to the common operator type by looking at all the argument types and finding the closest one that all the arguments can be cast to. When no common operator type is found the original expression will be returned and an Analysis Exception will be raised at the type checking phase.

    Definition Classes
    TypeCoercionBase
  42. object IntegralDivision extends Rule[LogicalPlan] with TypeCoercionRule

    The DIV operator always returns long-type value.

    The DIV operator always returns long-type value. This rule cast the integral inputs to long type, to avoid overflow during calculation.

    Definition Classes
    TypeCoercionBase
  43. object MapZipWithCoercion extends Rule[LogicalPlan] with TypeCoercionRule

    Coerces key types of two different MapType arguments of the MapZipWith expression to a common type.

    Coerces key types of two different MapType arguments of the MapZipWith expression to a common type.

    Definition Classes
    TypeCoercionBase
  44. object ProcedureArgumentCoercion extends Rule[LogicalPlan]

    A type coercion rule that implicitly casts procedure arguments to expected types.

    A type coercion rule that implicitly casts procedure arguments to expected types.

    Definition Classes
    TypeCoercionBase
  45. object StackCoercion extends Rule[LogicalPlan] with TypeCoercionRule

    Coerces NullTypes in the Stack expression to the column types of the corresponding positions.

    Coerces NullTypes in the Stack expression to the column types of the corresponding positions.

    Definition Classes
    TypeCoercionBase
  46. object StringLiteralCoercion extends Rule[LogicalPlan] with TypeCoercionRule

    A special rule to support string literal as the second argument of date_add/date_sub functions, to keep backward compatibility as a temporary workaround.

    A special rule to support string literal as the second argument of date_add/date_sub functions, to keep backward compatibility as a temporary workaround. TODO(SPARK-28589): implement ANSI type type coercion and handle string literals.

    Definition Classes
    TypeCoercionBase
  47. object UnpivotCoercion extends Rule[LogicalPlan]

    Widens the data types of the Unpivot values.

    Widens the data types of the Unpivot values.

    Definition Classes
    TypeCoercionBase
  48. object WidenSetOperationTypes extends Rule[LogicalPlan]

    Widens the data types of the children of Union/Except/Intersect.

    Widens the data types of the children of Union/Except/Intersect. 1. When ANSI mode is off: Loosely based on rules from "Hadoop: The Definitive Guide" 2nd edition, by Tom White

    The implicit conversion rules can be summarized as follows:

    • Any integral numeric type can be implicitly converted to a wider type.
    • All the integral numeric types, FLOAT, and (perhaps surprisingly) STRING can be implicitly converted to DOUBLE.
    • TINYINT, SMALLINT, and INT can all be converted to FLOAT.
    • BOOLEAN types cannot be converted to any other type.
    • Any integral numeric type can be implicitly converted to decimal type.
    • two different decimal types will be converted into a wider decimal type for both of them.
    • decimal type will be converted into double if there float or double together with it.

    All types when UNION-ed with strings will be promoted to strings. Other string conversions are handled by PromoteStrings.

    Widening types might result in loss of precision in the following cases:

    • IntegerType to FloatType
    • LongType to FloatType
    • LongType to DoubleType
    • DecimalType to Double

    2. When ANSI mode is on: The implicit conversion is determined by the closest common data type from the precedent lists from left and right child. See the comments of Object AnsiTypeCoercion for details.

    Definition Classes
    TypeCoercionBase
  49. object WindowFrameCoercion extends Rule[LogicalPlan] with TypeCoercionRule

    Cast WindowFrame boundaries to the type they operate upon.

    Cast WindowFrame boundaries to the type they operate upon.

    Definition Classes
    TypeCoercionBase
  50. object AnsiDateTimeOperationsTypeCoercion

    ANSI type coercion helper that matches against date-time expressions in order to type coerce children to expected types.

    ANSI type coercion helper that matches against date-time expressions in order to type coerce children to expected types.

    Definition Classes
    TypeCoercionHelper
  51. object CaseWhenTypeCoercion

    Type coercion helper that matches against a CaseWhen expression in order to type coerce different branches to a common type.

    Type coercion helper that matches against a CaseWhen expression in order to type coerce different branches to a common type.

    Definition Classes
    TypeCoercionHelper
  52. object ConcatTypeCoercion

    Type coercion helper that matches against Concat expressions in order to type coerce expression's children to expected types.

    Type coercion helper that matches against Concat expressions in order to type coerce expression's children to expected types.

    Definition Classes
    TypeCoercionHelper
  53. object DateTimeOperationsTypeCoercion

    Type coercion helper that matches against date-time expressions in order to type coerce children to expected types.

    Type coercion helper that matches against date-time expressions in order to type coerce children to expected types.

    Definition Classes
    TypeCoercionHelper
  54. object EltTypeCoercion

    Type coercion helper that matches against Elt expression in order to type coerce expression's children to expected types.

    Type coercion helper that matches against Elt expression in order to type coerce expression's children to expected types.

    Definition Classes
    TypeCoercionHelper
  55. object FunctionArgumentTypeCoercion

    Type coercion helper that matches against function expression in order to type coerce function argument types to expected types.

    Type coercion helper that matches against function expression in order to type coerce function argument types to expected types.

    Definition Classes
    TypeCoercionHelper
  56. object IfTypeCoercion

    Type coercion helper that matches against an If expression in order to type coerce different branches to a common type.

    Type coercion helper that matches against an If expression in order to type coerce different branches to a common type.

    Definition Classes
    TypeCoercionHelper
  57. object ImplicitTypeCoercion

    Type coercion helper that matches against expression in order to type coerce expression's input types to expected types.

    Type coercion helper that matches against expression in order to type coerce expression's input types to expected types.

    Definition Classes
    TypeCoercionHelper
  58. object InTypeCoercion

    Type coercion helper that matches agaist In and InSubquery expressions in order to type coerce LHS and RHS to expected types.

    Type coercion helper that matches agaist In and InSubquery expressions in order to type coerce LHS and RHS to expected types.

    Definition Classes
    TypeCoercionHelper
  59. object MapZipWithTypeCoercion

    Type coercion helper that matches against MapZipWith expressions in order to type coerce key types of input maps to a common type.

    Type coercion helper that matches against MapZipWith expressions in order to type coerce key types of input maps to a common type.

    Definition Classes
    TypeCoercionHelper
  60. object WindowFrameTypeCoercion

    Type coercion helper that matches against WindowFrameTypeCoercion expression in order to type coerce window boundaries to the type they operate upon.

    Type coercion helper that matches against WindowFrameTypeCoercion expression in order to type coerce window boundaries to the type they operate upon.

    Definition Classes
    TypeCoercionHelper

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from TypeCoercionBase

Inherited from TypeCoercionHelper

Inherited from AnyRef

Inherited from Any

Ungrouped