Packages

class ExpressionResolver extends TreeNodeResolver[Expression, Expression] with ProducesUnresolvedSubtree with ResolvesExpressionChildren with CoercesExpressionTypes

The ExpressionResolver is used by the Resolver during the analysis to resolve expressions.

The functions here generally traverse unresolved Expression nodes recursively, constructing and returning the resolved Expression nodes bottom-up. This is the primary entry point for implementing expression analysis, wherein the resolve method accepts a fully unresolved Expression and returns a fully resolved Expression in response with all data types and attribute reference ID assigned for valid requests. This resolver also takes responsibility to detect any errors in the initial SQL query or DataFrame and return appropriate error messages including precise parse locations wherever possible.

Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ExpressionResolver
  2. CoercesExpressionTypes
  3. ProducesUnresolvedSubtree
  4. ResolvesExpressionChildren
  5. TreeNodeResolver
  6. QueryErrorsBase
  7. DataTypeErrorsBase
  8. SQLConfHelper
  9. AnyRef
  10. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new ExpressionResolver(resolver: Resolver, functionResolution: FunctionResolution, planLogger: PlanLogger)

    resolver

    Resolver is passed from the parent to resolve other operators which are nested in expressions.

    functionResolution

    FunctionResolution to resolve function expressions.

    planLogger

    PlanLogger to log expression tree resolution events.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. val ansiTransformations: Transformations
    Attributes
    protected
    Definition Classes
    CoercesExpressionTypes
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  7. def coerceExpressionTypes(expression: Expression, expressionTreeTraversal: ExpressionTreeTraversal): Expression

    Coerces the expression types by applying necessary transformations on the expression and its children.

    Coerces the expression types by applying necessary transformations on the expression and its children. Because fixed-point sometimes resolves type coercion in multiple passes, we apply each provided transformation twice, cyclically, to ensure that types are resolved. For example in a query like:

    SELECT '1' + '1'

    fixed-point analyzer requires two passes to resolve types.

    In the end, we apply DefaultCollationTypeCoercion. See DefaultCollationTypeCoercion doc for more info.

    Additionally, we copy the tags and origin in case the call to this method didn't come from ExpressionResolver, where they are copied generically.

    Definition Classes
    CoercesExpressionTypes
  8. def conf: SQLConf

    The active config object within the current scope.

    The active config object within the current scope. See SQLConf.get for more information.

    Definition Classes
    SQLConfHelper
  9. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  10. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  11. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  12. def getExpressionIdAssigner: ExpressionIdAssigner
  13. def getExpressionResolutionContextStack: ArrayDeque[ExpressionResolutionContext]

    Get the expression resolution context stack.

  14. def getExpressionTreeTraversals: ExpressionTreeTraversalStack

    Get the expression tree traversal stack.

  15. def getLastInvalidExpressionsInTheContextOfOperator: Seq[Expression]

    Returns all invalid expressions in the context of the parent operator from the most recent expression tree resolution.

  16. def getLastReferencedAttributes: HashMap[ExprId, Attribute]

    Returns all attributes that have been referenced during the most recent expression tree resolution.

  17. def getLcaResolver: LateralColumnAliasResolver

    Get the LateralColumnAliasResolver to resolve lateral column references.

  18. def getNameScopes: NameScopeStack

    Get NameScopeStack bound to the used Resolver.

  19. def getQueryContext(context: QueryContext): Array[QueryContext]
    Definition Classes
    DataTypeErrorsBase
  20. def getSummary(sqlContext: QueryContext): String
    Definition Classes
    DataTypeErrorsBase
  21. def getTimezoneAwareExpressionResolver: TimezoneAwareExpressionResolver

    Get the TimezoneAwareExpressionResolver to resolve timezone-aware expressions.

  22. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  23. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  24. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  25. val nonAnsiTransformations: Transformations
    Attributes
    protected
    Definition Classes
    CoercesExpressionTypes
  26. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  27. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  28. def ordinalNumber(i: Int): String
    Definition Classes
    QueryErrorsBase
  29. def quoteByDefault(elem: String): String
    Attributes
    protected[sql]
    Definition Classes
    DataTypeErrorsBase
  30. def resolve(unresolvedExpression: Expression): Expression

    This method is an expression analysis entry point.

    This method is an expression analysis entry point. The method first checks if the expression has already been resolved (necessary because of partially-unresolved subtrees, see ProducesUnresolvedSubtree). If not already resolved, method takes an unresolved Expression and chooses the right resolve* method using pattern matching on the unresolvedExpression type. This pattern matching enumerates all the expression node types that are supported by the single-pass analysis. When developers introduce a new Expression type to the Catalyst, they should implement a corresponding resolve* method in the ExpressionResolver and add it to this pattern match list.

    resolve will be called recursively during the expression tree traversal eventually producing a fully resolved expression subtree or a descriptive error message.

    resolve can recursively call resolver to resolve nested operators (e.g. scalar subqueries):

    SELECT * FROM VALUES (1), (2) WHERE col1 IN (SELECT 1);

    In this case IN is an expression and SELECT 1 is a nested operator tree for which the ExpressionResolver would invoke the Resolver.

    This function avoids wrappers like CurrentOrigin.withOrigin to avoid deep recursion stacks, because expression trees may be quite deep.

    Definition Classes
    ExpressionResolver → TreeNodeResolver
  31. def resolveAggregateExpressions(sourceUnresolvedAggregateExpressions: Seq[NamedExpression], unresolvedAggregate: Aggregate): ResolvedAggregateExpressions

    Resolve aggregate expressions in Aggregate operator.

    Resolve aggregate expressions in Aggregate operator.

    The Aggregate list can contain different unresolved expressions before the resolution, which will be resolved using generic resolve. However, UnresolvedStar is a special case, because it is expanded into a sequence of NamedExpressions. Because of that this method returns a sequence and doesn't conform to generic resolve interface - it's called directly from the AggregateResolver during Aggregate resolution.

    Besides resolution, we do the following:

    • If there is a UnresolvedStar in the list we set hasStar to true in order to throw if there are any ordinals in grouping expressions. Example of an invalid query:
    SELECT * FROM VALUES(1) GROUP BY 1;
    • If there is an expression which has aggregate function in its subtree, we add it to the expressionsWithAggregateFunctions list in order to throw if there is any ordinal in grouping expressions which references that aggregate expression. Example of an invalid query:
    SELECT count(col1) FROM VALUES(1) GROUP BY 1;
    • If the resolved expression is an Alias, add it to scopes.current.topAggregateExpressionsByAliasName so it can be used for grouping expressions resolution, if needed. Example of a query with an Alias:
      1. Implicit alias:
    SELECT col1 + col2 FROM VALUES(1, 2) GROUP BY `(col1 + col2)`;

    2. Explicit alias:

    SELECT 1 AS column GROUP BY column;

    While resolving the list, we have to keep track of all the expressions that don't have AggregateExpressions in their subtrees (expressionsWithoutAggregates) and whether any of aggregate expressions (that are not expressionsWithoutAggregates) has attributes in the subtree outside of AggregateExpressionss (hasAttributeOutsideOfAggregateExpressions). This is used when resolving GROUP BY ALL in the AggregateResolver.resolveGroupByAll.

  32. def resolveExpressionTreeInOperator(unresolvedExpression: Expression, parentOperator: LogicalPlan): Expression

    Resolve unresolvedExpression which is a child of parentOperator.

    Resolve unresolvedExpression which is a child of parentOperator. This is the main entry point into the ExpressionResolver for operators.

  33. def resolveGroupingExpressions(sourceUnresolvedGroupingExpressions: Seq[Expression], unresolvedAggregate: Aggregate): Seq[Expression]

    Resolve grouping expressions in Aggregate operator.

    Resolve grouping expressions in Aggregate operator.

    It's done for every expression using the resolveExpressionTreeInOperatorImpl. For cases where grouping is done based on aliases the resolution is following:

    • If the expression can be resolved using the child's output (scopes.current.output), resolve it that way. Example:
    SELECT col1 FROM VALUES(1) GROUP BY `col1`;
    • If not, try to resolve it as a top level Alias (which was populated during the resolution of the aggregate expressions). Example:
      1. Group by implicit alias
    SELECT concat_ws(' ', 'a', 'b') GROUP BY `concat_ws( , a, b)`;

    2. Group by explicit alias

    SELECT col1 AS column_1 FROM VALUES(1) GROUP BY column_1;

    After resolving the expression, remove the top level Alias if it exists.

  34. def resolveLimitLikeExpression(unresolvedLimitLikeExpr: Expression, partiallyResolvedLimitLike: LogicalPlan): Expression

    Resolve and validate the limit like expressions from either LocalLimit, GlobalLimit, Offset or Tail operator.

  35. def resolveProjectList(sourceUnresolvedProjectList: Seq[NamedExpression], operator: LogicalPlan): ResolvedProjectList

    The Project list can contain different unresolved expressions before the resolution, which will be resolved using generic resolve.

    The Project list can contain different unresolved expressions before the resolution, which will be resolved using generic resolve. However, UnresolvedStar is a special case, because it is expanded into a sequence of NamedExpressions. Because of that this method returns a sequence and doesn't conform to generic resolve interface - it's called directly from the Resolver during Project resolution.

    The output sequence can be larger than the input sequence due to UnresolvedStar expansion.

    returns

    The list of resolved expressions along with flags indicating whether the resolved project list contains aggregate expressions or attributes (encapsulated in ResolvedProjectList) which are used during the further resolution of the tree. The following query:

    SELECT COUNT(col1), 2 FROM VALUES(1);

    would have a project list with two expressions: COUNT(col1) and 2. After the resolution it would return the following result: ResolvedProjectList( expressions = [count(col1) as count(col1), 2 AS 2], hasAggregateExpressions = true, // because it contains count(col1) in the project list hasLateralColumnAlias = false // because there are no lateral column aliases )

  36. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  37. def toDSOption(option: String): String
    Definition Classes
    DataTypeErrorsBase
  38. def toSQLConf(conf: String): String
    Definition Classes
    DataTypeErrorsBase
  39. def toSQLConfVal(conf: String): String
    Definition Classes
    QueryErrorsBase
  40. def toSQLExpr(e: Expression): String
    Definition Classes
    QueryErrorsBase
  41. def toSQLId(parts: Seq[String]): String
    Definition Classes
    DataTypeErrorsBase
  42. def toSQLId(parts: String): String
    Definition Classes
    DataTypeErrorsBase
  43. def toSQLStmt(text: String): String
    Definition Classes
    DataTypeErrorsBase
  44. def toSQLType(t: AbstractDataType): String
    Definition Classes
    DataTypeErrorsBase
  45. def toSQLType(text: String): String
    Definition Classes
    DataTypeErrorsBase
  46. def toSQLValue(v: Any, t: DataType): String
    Definition Classes
    QueryErrorsBase
  47. def toSQLValue(value: Double): String
    Definition Classes
    DataTypeErrorsBase
  48. def toSQLValue(value: Float): String
    Definition Classes
    DataTypeErrorsBase
  49. def toSQLValue(value: Long): String
    Definition Classes
    DataTypeErrorsBase
  50. def toSQLValue(value: Int): String
    Definition Classes
    DataTypeErrorsBase
  51. def toSQLValue(value: Short): String
    Definition Classes
    DataTypeErrorsBase
  52. def toSQLValue(value: UTF8String): String
    Definition Classes
    DataTypeErrorsBase
  53. def toSQLValue(value: String): String
    Definition Classes
    DataTypeErrorsBase
  54. def toString(): String
    Definition Classes
    AnyRef → Any
  55. def tryPopSinglePassSubtreeBoundary(unresolvedExpression: Expression): Boolean

    Try to pop the tag that marks the boundary of the single-pass subtree resolution.

    Try to pop the tag that marks the boundary of the single-pass subtree resolution. ExpressionResolver calls this method to check if the subtree traversal needs to be stopped because lower subtree is already resolved.

    Attributes
    protected
    Definition Classes
    ProducesUnresolvedSubtree
  56. def validateExpressionUnderSupportedOperator(expression: Expression): Unit

    Validate if expression is under supported operator or not.

    Validate if expression is under supported operator or not. In case it's not, add expression to the ExpressionTreeTraversal.invalidExpressionsInTheContextOfOperator list to throw error later, when getLastInvalidExpressionsInTheContextOfOperator is called by the Resolver. Here, we avoid adding AggregateExpressions when they are under a Sort or Filter on top of Aggregate as they are not transformed to attributes at the moment. Please see UnsupportedExpressionInOperatorValidation.isExpressionInUnsupportedOperator for more info.

  57. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  58. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  59. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  60. def withResolvedChildren(unresolvedExpression: Expression, resolveChild: (Expression) => Expression): Expression

    Resolves generic Expression children and returns its copy with children resolved.

    Resolves generic Expression children and returns its copy with children resolved.

    We avoid extra method calls here, because resolveChild usually recurses deeply into resolution, and expression trees may be quite deep.

    Attributes
    protected
    Definition Classes
    ResolvesExpressionChildren
  61. def withResolvedChildren(unresolvedExpression: QuaternaryExpression, resolveChild: (Expression) => Expression): Expression

    Resolves QuaternaryExpression children and returns its copy with children resolved.

    Resolves QuaternaryExpression children and returns its copy with children resolved.

    Attributes
    protected
    Definition Classes
    ResolvesExpressionChildren
  62. def withResolvedChildren(unresolvedExpression: TernaryExpression, resolveChild: (Expression) => Expression): Expression

    Resolves TernaryExpression children and returns its copy with children resolved.

    Resolves TernaryExpression children and returns its copy with children resolved.

    Attributes
    protected
    Definition Classes
    ResolvesExpressionChildren
  63. def withResolvedChildren(unresolvedExpression: BinaryExpression, resolveChild: (Expression) => Expression): Expression

    Resolves BinaryExpression children and returns its copy with children resolved.

    Resolves BinaryExpression children and returns its copy with children resolved.

    Attributes
    protected
    Definition Classes
    ResolvesExpressionChildren
  64. def withResolvedChildren(unresolvedExpression: UnaryExpression, resolveChild: (Expression) => Expression): Expression

    Resolves UnaryExpression children and returns its copy with children resolved.

    Resolves UnaryExpression children and returns its copy with children resolved.

    Attributes
    protected
    Definition Classes
    ResolvesExpressionChildren
  65. def withResolvedSubtree(expression: Expression, expressionResolver: (Expression) => Expression)(body: => Expression): Expression

    Helper method used to resolve a subtree that is generated as part of the resolution of some node.

    Helper method used to resolve a subtree that is generated as part of the resolution of some node. Method ensures that the downwards traversal never visits previously resolved nodes by tracking the limits of the traversal with a tag. Invokes a resolver callback to resolve children, but DOES NOT resolve the root of the subtree.

    If the result of the callback is the same object as the source expression, we don't perform the downwards traversal. This is both more optimal and a fail-safe mechanism in case we accidentally lose the ResolverTag.SINGLE_PASS_SUBTREE_BOUNDARY tag.

    Attributes
    protected
    Definition Classes
    ProducesUnresolvedSubtree
  66. def withSQLConf[T](pairs: (String, String)*)(f: => T): T

    Sets all SQL configurations specified in pairs, calls f, and then restores all SQL configurations.

    Sets all SQL configurations specified in pairs, calls f, and then restores all SQL configurations.

    Attributes
    protected
    Definition Classes
    SQLConfHelper

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from CoercesExpressionTypes

Inherited from QueryErrorsBase

Inherited from DataTypeErrorsBase

Inherited from SQLConfHelper

Inherited from AnyRef

Inherited from Any

Ungrouped