class ExpressionResolver extends TreeNodeResolver[Expression, Expression] with ProducesUnresolvedSubtree with ResolvesExpressionChildren with CoercesExpressionTypes
The ExpressionResolver is used by the Resolver during the analysis to resolve expressions.
The functions here generally traverse unresolved Expression nodes recursively, constructing and returning the resolved Expression nodes bottom-up. This is the primary entry point for implementing expression analysis, wherein the resolve method accepts a fully unresolved Expression and returns a fully resolved Expression in response with all data types and attribute reference ID assigned for valid requests. This resolver also takes responsibility to detect any errors in the initial SQL query or DataFrame and return appropriate error messages including precise parse locations wherever possible.
- Alphabetic
- By Inheritance
- ExpressionResolver
- CoercesExpressionTypes
- ProducesUnresolvedSubtree
- ResolvesExpressionChildren
- TreeNodeResolver
- QueryErrorsBase
- DataTypeErrorsBase
- SQLConfHelper
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new ExpressionResolver(resolver: Resolver, functionResolution: FunctionResolution, planLogger: PlanLogger)
- resolver
Resolver is passed from the parent to resolve other operators which are nested in expressions.
- functionResolution
FunctionResolution to resolve function expressions.
- planLogger
PlanLogger to log expression tree resolution events.
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- val ansiTransformations: Transformations
- Attributes
- protected
- Definition Classes
- CoercesExpressionTypes
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- def coerceExpressionTypes(expression: Expression, expressionTreeTraversal: ExpressionTreeTraversal): Expression
Coerces the expression types by applying necessary transformations on the expression and its children.
Coerces the expression types by applying necessary transformations on the expression and its children. Because fixed-point sometimes resolves type coercion in multiple passes, we apply each provided transformation twice, cyclically, to ensure that types are resolved. For example in a query like:
SELECT '1' + '1'
fixed-point analyzer requires two passes to resolve types.
In the end, we apply DefaultCollationTypeCoercion. See DefaultCollationTypeCoercion doc for more info.
Additionally, we copy the tags and origin in case the call to this method didn't come from ExpressionResolver, where they are copied generically.
- Definition Classes
- CoercesExpressionTypes
- def conf: SQLConf
The active config object within the current scope.
The active config object within the current scope. See SQLConf.get for more information.
- Definition Classes
- SQLConfHelper
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def getExpressionIdAssigner: ExpressionIdAssigner
- def getExpressionResolutionContextStack: ArrayDeque[ExpressionResolutionContext]
Get the expression resolution context stack.
- def getExpressionTreeTraversals: ExpressionTreeTraversalStack
Get the expression tree traversal stack.
- def getLastInvalidExpressionsInTheContextOfOperator: Seq[Expression]
Returns all invalid expressions in the context of the parent operator from the most recent expression tree resolution.
- def getLastReferencedAttributes: HashMap[ExprId, Attribute]
Returns all attributes that have been referenced during the most recent expression tree resolution.
- def getLcaResolver: LateralColumnAliasResolver
Get the LateralColumnAliasResolver to resolve lateral column references.
- def getNameScopes: NameScopeStack
Get NameScopeStack bound to the used Resolver.
- def getQueryContext(context: QueryContext): Array[QueryContext]
- Definition Classes
- DataTypeErrorsBase
- def getSummary(sqlContext: QueryContext): String
- Definition Classes
- DataTypeErrorsBase
- def getTimezoneAwareExpressionResolver: TimezoneAwareExpressionResolver
Get the TimezoneAwareExpressionResolver to resolve timezone-aware expressions.
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- val nonAnsiTransformations: Transformations
- Attributes
- protected
- Definition Classes
- CoercesExpressionTypes
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- def ordinalNumber(i: Int): String
- Definition Classes
- QueryErrorsBase
- def quoteByDefault(elem: String): String
- Attributes
- protected[sql]
- Definition Classes
- DataTypeErrorsBase
- def resolve(unresolvedExpression: Expression): Expression
This method is an expression analysis entry point.
This method is an expression analysis entry point. The method first checks if the expression has already been resolved (necessary because of partially-unresolved subtrees, see ProducesUnresolvedSubtree). If not already resolved, method takes an unresolved Expression and chooses the right
resolve*method using pattern matching on theunresolvedExpressiontype. This pattern matching enumerates all the expression node types that are supported by the single-pass analysis. When developers introduce a new Expression type to the Catalyst, they should implement a correspondingresolve*method in the ExpressionResolver and add it to this pattern match list.resolve will be called recursively during the expression tree traversal eventually producing a fully resolved expression subtree or a descriptive error message.
resolve can recursively call
resolverto resolve nested operators (e.g. scalar subqueries):SELECT * FROM VALUES (1), (2) WHERE col1 IN (SELECT 1);
In this case
INis an expression andSELECT 1is a nested operator tree for which the ExpressionResolver would invoke the Resolver.This function avoids wrappers like CurrentOrigin.withOrigin to avoid deep recursion stacks, because expression trees may be quite deep.
- Definition Classes
- ExpressionResolver → TreeNodeResolver
- def resolveAggregateExpressions(sourceUnresolvedAggregateExpressions: Seq[NamedExpression], unresolvedAggregate: Aggregate): ResolvedAggregateExpressions
Resolve aggregate expressions in Aggregate operator.
Resolve aggregate expressions in Aggregate operator.
The Aggregate list can contain different unresolved expressions before the resolution, which will be resolved using generic resolve. However, UnresolvedStar is a special case, because it is expanded into a sequence of NamedExpressions. Because of that this method returns a sequence and doesn't conform to generic resolve interface - it's called directly from the AggregateResolver during Aggregate resolution.
Besides resolution, we do the following:
- If there is a UnresolvedStar in the list we set
hasStarto true in order to throw if there are any ordinals in grouping expressions. Example of an invalid query:
SELECT * FROM VALUES(1) GROUP BY 1;
- If there is an expression which has aggregate function in its subtree, we add it to the
expressionsWithAggregateFunctionslist in order to throw if there is any ordinal in grouping expressions which references that aggregate expression. Example of an invalid query:
SELECT count(col1) FROM VALUES(1) GROUP BY 1;
- If the resolved expression is an Alias, add it to
scopes.current.topAggregateExpressionsByAliasNameso it can be used for grouping expressions resolution, if needed. Example of a query with an Alias:- Implicit alias:
SELECT col1 + col2 FROM VALUES(1, 2) GROUP BY `(col1 + col2)`;
2. Explicit alias:
SELECT 1 AS column GROUP BY column;While resolving the list, we have to keep track of all the expressions that don't have AggregateExpressions in their subtrees (
expressionsWithoutAggregates) and whether any of aggregate expressions (that are notexpressionsWithoutAggregates) has attributes in the subtree outside of AggregateExpressionss (hasAttributeOutsideOfAggregateExpressions). This is used when resolvingGROUP BY ALLin the AggregateResolver.resolveGroupByAll. - If there is a UnresolvedStar in the list we set
- def resolveExpressionTreeInOperator(unresolvedExpression: Expression, parentOperator: LogicalPlan): Expression
Resolve
unresolvedExpressionwhich is a child ofparentOperator.Resolve
unresolvedExpressionwhich is a child ofparentOperator. This is the main entry point into the ExpressionResolver for operators. - def resolveGroupingExpressions(sourceUnresolvedGroupingExpressions: Seq[Expression], unresolvedAggregate: Aggregate): Seq[Expression]
Resolve grouping expressions in Aggregate operator.
Resolve grouping expressions in Aggregate operator.
It's done for every expression using the
resolveExpressionTreeInOperatorImpl. For cases where grouping is done based on aliases the resolution is following:- If the expression can be resolved using the child's output (
scopes.current.output), resolve it that way. Example:
SELECT col1 FROM VALUES(1) GROUP BY `col1`;- If not, try to resolve it as a top level Alias (which was populated during the
resolution of the aggregate expressions).
Example:
- Group by implicit alias
SELECT concat_ws(' ', 'a', 'b') GROUP BY `concat_ws( , a, b)`;
2. Group by explicit alias
SELECT col1 AS column_1 FROM VALUES(1) GROUP BY column_1;After resolving the expression, remove the top level Alias if it exists.
- If the expression can be resolved using the child's output (
- def resolveLimitLikeExpression(unresolvedLimitLikeExpr: Expression, partiallyResolvedLimitLike: LogicalPlan): Expression
Resolve and validate the limit like expressions from either LocalLimit, GlobalLimit, Offset or Tail operator.
- def resolveProjectList(sourceUnresolvedProjectList: Seq[NamedExpression], operator: LogicalPlan): ResolvedProjectList
The Project list can contain different unresolved expressions before the resolution, which will be resolved using generic resolve.
The Project list can contain different unresolved expressions before the resolution, which will be resolved using generic resolve. However, UnresolvedStar is a special case, because it is expanded into a sequence of NamedExpressions. Because of that this method returns a sequence and doesn't conform to generic resolve interface - it's called directly from the Resolver during Project resolution.
The output sequence can be larger than the input sequence due to UnresolvedStar expansion.
- returns
The list of resolved expressions along with flags indicating whether the resolved project list contains aggregate expressions or attributes (encapsulated in ResolvedProjectList) which are used during the further resolution of the tree. The following query:
SELECT COUNT(col1), 2 FROM VALUES(1);
would have a project list with two expressions:
COUNT(col1)and2. After the resolution it would return the following result: ResolvedProjectList( expressions = [count(col1) as count(col1), 2 AS 2], hasAggregateExpressions = true, // because it containscount(col1)in the project list hasLateralColumnAlias = false // because there are no lateral column aliases )
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toDSOption(option: String): String
- Definition Classes
- DataTypeErrorsBase
- def toSQLConf(conf: String): String
- Definition Classes
- DataTypeErrorsBase
- def toSQLConfVal(conf: String): String
- Definition Classes
- QueryErrorsBase
- def toSQLExpr(e: Expression): String
- Definition Classes
- QueryErrorsBase
- def toSQLId(parts: Seq[String]): String
- Definition Classes
- DataTypeErrorsBase
- def toSQLId(parts: String): String
- Definition Classes
- DataTypeErrorsBase
- def toSQLStmt(text: String): String
- Definition Classes
- DataTypeErrorsBase
- def toSQLType(t: AbstractDataType): String
- Definition Classes
- DataTypeErrorsBase
- def toSQLType(text: String): String
- Definition Classes
- DataTypeErrorsBase
- def toSQLValue(v: Any, t: DataType): String
- Definition Classes
- QueryErrorsBase
- def toSQLValue(value: Double): String
- Definition Classes
- DataTypeErrorsBase
- def toSQLValue(value: Float): String
- Definition Classes
- DataTypeErrorsBase
- def toSQLValue(value: Long): String
- Definition Classes
- DataTypeErrorsBase
- def toSQLValue(value: Int): String
- Definition Classes
- DataTypeErrorsBase
- def toSQLValue(value: Short): String
- Definition Classes
- DataTypeErrorsBase
- def toSQLValue(value: UTF8String): String
- Definition Classes
- DataTypeErrorsBase
- def toSQLValue(value: String): String
- Definition Classes
- DataTypeErrorsBase
- def toString(): String
- Definition Classes
- AnyRef → Any
- def tryPopSinglePassSubtreeBoundary(unresolvedExpression: Expression): Boolean
Try to pop the tag that marks the boundary of the single-pass subtree resolution.
Try to pop the tag that marks the boundary of the single-pass subtree resolution. ExpressionResolver calls this method to check if the subtree traversal needs to be stopped because lower subtree is already resolved.
- Attributes
- protected
- Definition Classes
- ProducesUnresolvedSubtree
- def validateExpressionUnderSupportedOperator(expression: Expression): Unit
Validate if
expressionis under supported operator or not.Validate if
expressionis under supported operator or not. In case it's not, addexpressionto the ExpressionTreeTraversal.invalidExpressionsInTheContextOfOperator list to throw error later, when getLastInvalidExpressionsInTheContextOfOperator is called by the Resolver. Here, we avoid adding AggregateExpressions when they are under a Sort or Filter on top of Aggregate as they are not transformed to attributes at the moment. Please see UnsupportedExpressionInOperatorValidation.isExpressionInUnsupportedOperator for more info. - final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- def withResolvedChildren(unresolvedExpression: Expression, resolveChild: (Expression) => Expression): Expression
Resolves generic Expression children and returns its copy with children resolved.
Resolves generic Expression children and returns its copy with children resolved.
We avoid extra method calls here, because
resolveChildusually recurses deeply into resolution, and expression trees may be quite deep.- Attributes
- protected
- Definition Classes
- ResolvesExpressionChildren
- def withResolvedChildren(unresolvedExpression: QuaternaryExpression, resolveChild: (Expression) => Expression): Expression
Resolves QuaternaryExpression children and returns its copy with children resolved.
Resolves QuaternaryExpression children and returns its copy with children resolved.
- Attributes
- protected
- Definition Classes
- ResolvesExpressionChildren
- def withResolvedChildren(unresolvedExpression: TernaryExpression, resolveChild: (Expression) => Expression): Expression
Resolves TernaryExpression children and returns its copy with children resolved.
Resolves TernaryExpression children and returns its copy with children resolved.
- Attributes
- protected
- Definition Classes
- ResolvesExpressionChildren
- def withResolvedChildren(unresolvedExpression: BinaryExpression, resolveChild: (Expression) => Expression): Expression
Resolves BinaryExpression children and returns its copy with children resolved.
Resolves BinaryExpression children and returns its copy with children resolved.
- Attributes
- protected
- Definition Classes
- ResolvesExpressionChildren
- def withResolvedChildren(unresolvedExpression: UnaryExpression, resolveChild: (Expression) => Expression): Expression
Resolves UnaryExpression children and returns its copy with children resolved.
Resolves UnaryExpression children and returns its copy with children resolved.
- Attributes
- protected
- Definition Classes
- ResolvesExpressionChildren
- def withResolvedSubtree(expression: Expression, expressionResolver: (Expression) => Expression)(body: => Expression): Expression
Helper method used to resolve a subtree that is generated as part of the resolution of some node.
Helper method used to resolve a subtree that is generated as part of the resolution of some node. Method ensures that the downwards traversal never visits previously resolved nodes by tracking the limits of the traversal with a tag. Invokes a resolver callback to resolve children, but DOES NOT resolve the root of the subtree.
If the result of the callback is the same object as the source
expression, we don't perform the downwards traversal. This is both more optimal and a fail-safe mechanism in case we accidentally lose the ResolverTag.SINGLE_PASS_SUBTREE_BOUNDARY tag.- Attributes
- protected
- Definition Classes
- ProducesUnresolvedSubtree
- def withSQLConf[T](pairs: (String, String)*)(f: => T): T
Sets all SQL configurations specified in
pairs, callsf, and then restores all SQL configurations.Sets all SQL configurations specified in
pairs, callsf, and then restores all SQL configurations.- Attributes
- protected
- Definition Classes
- SQLConfHelper
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)