resolver

package resolver

Linear Supertypes

AnyRef, Any

Ordering

Alphabetic
By Inheritance

Inherited

resolver
AnyRef
Any

Hide All
Show All

Visibility

Public
Protected

Type Members

class AggregateExpressionResolver extends TreeNodeResolver[AggregateExpression, Expression] with ResolvesExpressionChildren
A resolver for AggregateExpressions which are introduced while resolving an UnresolvedFunction.
A resolver for AggregateExpressions which are introduced while resolving an UnresolvedFunction. It is responsible for the following:
- Handling of the exceptions related to AggregateExpressions.
- Updating the ExpressionResolver.expressionResolutionContextStack.
- Applying type coercion rules to the AggregateExpressionss children. This is the only resolution that we apply here as we already resolved the children of AggregateExpression in the FunctionResolver.
class AliasResolver extends TreeNodeResolver[UnresolvedAlias, Expression] with ResolvesExpressionChildren
Resolver class that resolves unresolved aliases and handles user-specified aliases.
case class AnalyzerBridgeState(relationsWithResolvedMetadata: RelationsWithResolvedMetadata = new AnalyzerBridgeState.RelationsWithResolvedMetadata) extends Product with Serializable
The AnalyzerBridgeState is a state passed from legacy Analyzer to the single-pass Resolver.
The AnalyzerBridgeState is a state passed from legacy Analyzer to the single-pass Resolver.
relationsWithResolvedMetadata
A map from UnresolvedRelation to the relations with resolved metadata. It allows us to reuse the relation metadata and avoid duplicate catalog/table lookups in dual-run mode (when ANALYZER_SINGLE_PASS_RESOLVER_RELATION_BRIDGING_ENABLED is true).
class AttributeScopeStack extends AnyRef
The AttributeScopeStack is used to validate that the attribute which was encountered by the ExpressionResolutionValidator is in the current operator's visibility scope.
The AttributeScopeStack is used to validate that the attribute which was encountered by the ExpressionResolutionValidator is in the current operator's visibility scope. We use AttributeSet as scope implementation here to check the equality of attributes based on their expression IDs.
E.g. for the following SQL query:
```
SELECT a, a, a + col2 FROM (SELECT col1 as a, col2 FROM VALUES (1, 2));
```
Having the following logical plan:
```
Project [a#2, a#2, (a#2 + col2#1) AS (a + col2)#3]
+- SubqueryAlias __auto_generated_subquery_name
   +- Project [col1#0 AS a#2, col2#1]
      +- LocalRelation [col1#0, col2#1]
```
The LocalRelation outputs attributes with IDs #0 and #1, which can be referenced by the lower Project. This Project produces a new attribute ID #2 for an alias and retains the old ID #1 for col2. The upper Project references a twice using the same ID #2 and produces a new ID #3 for an alias of a + col2.
class BinaryArithmeticResolver extends TreeNodeResolver[BinaryArithmetic, Expression] with ProducesUnresolvedSubtree
BinaryArithmeticResolver is invoked by ExpressionResolver in order to resolve BinaryArithmetic nodes.
BinaryArithmeticResolver is invoked by ExpressionResolver in order to resolve BinaryArithmetic nodes. During resolution, calling BinaryArithmeticWithDatetimeResolver and applying type coercion can result in BinaryArithmetic producing some other type of node or a subtree of nodes. In such cases a downwards traversal is necessary, but not going deeper than the original expression's children, since all nodes below that point are guaranteed to be already resolved.
For example, given a query:
SELECT '4 11:11' - INTERVAL '4 22:12' DAY TO MINUTE
BinaryArithmeticResolver is called for the following expression:
Subtract( Literal('4 11:11', StringType), Literal(Interval('4 22:12' DAY TO MINUTE), DayTimeIntervalType(0,2)) )
After calling BinaryArithmeticWithDatetimeResolver and applying type coercion, the expression is transformed into:
Cast( DatetimeSub( TimeAdd( Literal('4 11:11', StringType), UnaryMinus( Literal(Interval('4 22:12' DAY TO MINUTE), DayTimeIntervalType(0,2)) ) ) ) )
A single Subtract node is replaced with a subtree of nodes. In order to resolve this subtree we need to invoke ExpressionResolver recursively on the top-most node's children. The top-most node itself is not resolved recursively in order to avoid recursive calls to BinaryArithmeticResolver and other sub-resolvers. To prevent a case where we resolve the same node twice, we need to mark nodes that will act as a limit for the downwards traversal by applying a ExpressionResolver.SINGLE_PASS_SUBTREE_BOUNDARY tag to them. These children along with all the nodes below them are guaranteed to be resolved at this point. When ExpressionResolver reaches one of the tagged nodes, it returns identity rather than resolving it. Finally, after resolving the subtree, we need to resolve the top-most node itself, which in this case means applying a timezone, if necessary.
class BridgedRelationMetadataProvider extends RelationMetadataProvider
The BridgedRelationMetadataProvider is a RelationMetadataProvider that just reuses resolved metadata from the AnalyzerBridgeState.
The BridgedRelationMetadataProvider is a RelationMetadataProvider that just reuses resolved metadata from the AnalyzerBridgeState. This is used in the single-pass Resolver to avoid duplicate catalog/table lookups in dual-run mode, so metadata is simply reused from the fixed-point Analyzer run. We strictly rely on the AnalyzerBridgeState to avoid any blocking calls here.
class ConditionalExpressionResolver extends TreeNodeResolver[ConditionalExpression, Expression] with ResolvesExpressionChildren
Resolver for If, CaseWhen and Coalesce expressions.
class CreateNamedStructResolver extends TreeNodeResolver[CreateNamedStruct, Expression] with ResolvesExpressionChildren
Resolves CreateNamedStruct nodes by recursively resolving children.
Resolves CreateNamedStruct nodes by recursively resolving children. If CreateNamedStruct is not directly under an Alias, removes aliases from struct fields. Otherwise, let AliasResolver handle the removal.
class CteRegistry extends AnyRef
The CteRegistry is responsible for managing the stack of CteScopes and resolving visible CTERelationDef names.
class CteScope extends AnyRef
The CteScope is responsible for keeping track of visible and known CTE definitions at a given stage of a SQL query/DataFrame program resolution.
The CteScope is responsible for keeping track of visible and known CTE definitions at a given stage of a SQL query/DataFrame program resolution. These scopes are stacked and the stack is managed by the CteRegistry. The scope is created per single WITH clause.
The CTE operators are:
- UnresolvedWith. This is a host operator that contains a list of unresolved CTE definitions from the WITH clause and a single child operator, which is the actual unresolved SELECT query.
- UnresolvedRelation. This is a generic unresolved relation operator that will sometimes be resolved to a CTE definition and later replaced with a CTERelationRef. The CTE takes precedence over a regular table or a view when resolving this identifier.
- CTERelationDef. This is a reusable logical plan, which will later be referenced by the lower CTE definitions and UnresolvedWith child.
- CTERelationRef. This is a leaf node similar to a relation operator that references a certain CTERelationDef by its ID. It has a name (unique locally for a WITH clause list) and an ID (unique for all the CTEs in a query).
- WithCTE. This is a host operator that contains a list of resolved CTE definitions from the WITH clause and a single child operator, which is the actual resolved SELECT query.
The task of the Resolver is to correctly place WithCTE with CTERelationDefs inside and make sure that CTERelationRefs correctly reference CTERelationDefs with their IDs. The decision whether to inline those CTE subtrees or not is made by the Optimizer, unlike what Spark does for the Views (always inline during the analysis).
There are some caveats in how Spark places those operators and resolves their names:
- Ambiguous CTE definition names are disallowed only within a single WITH clause, and this is validated by the Parser in AstBuilder using QueryParsingErrors.duplicateCteDefinitionNamesError:
```
-- This is disallowed.
WITH cte AS (SELECT 1),
cte AS (SELECT 2)
SELECT * FROM cte;
```
- When UnresolvedRelation identifier is resolved to a CTERelationDef and there is a name conflict on several layers of CTE definitions, the lower definitions take precedence:
```
-- The result is `3`, lower [[CTERelationDef]] takes precedence.
WITH cte AS (
  SELECT 1
)
SELECT * FROM (
  WITH cte AS (
    SELECT 2
  )
  SELECT * FROM (
    WITH cte AS (
      SELECT 3
    )
    SELECT * FROM cte
  )
)
```
- Any subquery can contain UnresolvedWith on top of it, but WithCTE is not gonna be 1 to 1 to its unresolved counterpart. For example, if we are dealing with simple subqueries, CTERelationDefs will be merged together under a single WithCTE. The previous example would produce the following resolved plan:
```
WithCTE
:- CTERelationDef 18, false
:  +- ...
:- CTERelationDef 19, false
:  +- ...
:- CTERelationDef 20, false
:  +- ...
+- Project [3#1203]
:  +- ...
```
- However, if we have any expression subquery (scalar/IN/EXISTS...), the top CTERelationDefs and subquery's CTERelationDef won't be merged together (as they are separated by an expression tree):
```
WITH cte AS (
  SELECT 1 AS col1
)
SELECT * FROM cte WHERE col1 IN (
  WITH cte AS (
    SELECT 2
  )
  SELECT * FROM cte
)
```
->
```
WithCTE
:- CTERelationDef 21, false
:  +- ...
+- Project [col1#1223]
   +- Filter col1#1223 IN (list#1222 [])
      :  +- WithCTE
      :     :- CTERelationDef 22, false
      :     :  +- ...
      :     +- Project [2#1241]
      :        +- ...
      +- ...
```
- Upper CTEs are visible through subqueries and can be referenced by lower operators, but not through the View boundary:
```
CREATE VIEW v1 AS SELECT 1;
CREATE VIEW v2 AS SELECT * FROM v1;

-- The result is 1.
-- The `v2` body will be inlined in the main query tree during the analysis, but upper `v1`
-- CTE definition _won't_ take precedence over the lower `v1` view.
WITH v1 AS (
  SELECT 2
)
SELECT * FROM v2;
```
trait DelegatesResolutionToExtensions extends AnyRef
The DelegatesResolutionToExtensions is a trait which provides a method to delegate the resolution of unresolved operators to a list of ResolverExtensions.
class ExplicitlyUnsupportedResolverFeature extends Exception
This is an addon to ResolverGuard functionality for features that cannot be determined by only looking at the unresolved plan.
This is an addon to ResolverGuard functionality for features that cannot be determined by only looking at the unresolved plan. Resolver will throw this control-flow exception when it encounters some explicitly unsupported feature. Later behavior depends on the value of HybridAnalyzer.checkSupportedSinglePassFeatures flag:
- If it is true: It will later be caught by HybridAnalyzer to abort single-pass analysis without comparing single-pass and fixed-point results. The motivation for this feature is the same as for the ResolverGuard - we want to have an explicit allowlist of unimplemented features that we are aware of, and UNSUPPORTED_SINGLE_PASS_ANALYZER_FEATURE will signal us the rest of the gaps.
- If it is false: It will be thrown by the HybridAnalyzer in order to get better sense of coverage.
For example, UnresolvedRelation can be intermediately resolved by ResolveRelations as UnresolvedCatalogRelation or a View (among all others). Say that for now the views are not implemented, and we are aware of that, so ExplicitlyUnsupportedResolverFeature will be thrown in the middle of the single-pass analysis to abort it.
class ExpressionIdAssigner extends AnyRef
ExpressionIdAssigner is used by the ExpressionResolver to assign unique expression IDs to NamedExpressions (AttributeReferences and Aliases).
ExpressionIdAssigner is used by the ExpressionResolver to assign unique expression IDs to NamedExpressions (AttributeReferences and Aliases). This is necessary to ensure that Optimizer performs its work correctly and does not produce correctness issues.
The framework works the following way:
- Each leaf operator must have unique output IDs (even if it's the same table, view, or CTE).
- The AttributeReferences get propagated "upwards" through the operator tree with their IDs preserved.
- Each Alias gets assigned a new unique ID and it sticks with it after it gets converted to an AttributeReference when it is outputted from the operator that produced it.
- Any operator may have AttributeReferences with the same IDs in its output given it is the same attribute. Thus, **no multi-child operator may have children with conflicting AttributeReference IDs**. In other words, two subtrees must not output the AttributeReferences with the same IDs, since relations, views and CTEs all output unique attributes, and Aliases get assigned new IDs as well. ExpressionIdAssigner.assertOutputsHaveNoConflictingExpressionIds is used to assert this invariant.
For SQL queries, this framework provides correctness just by reallocating relation outputs and by validating the invariants mentioned above. Reallocation is done in Resolver.handleLeafOperator. If all the relations (even if it's the same table) have unique output IDs, the expression ID assignment will be correct, because there are no duplicate IDs in a pure unresolved tree. The old ID -> new ID mapping is not needed in this case. For example, consider this query:
```
SELECT * FROM t AS t1 CROSS JOIN t AS t2 ON t1.col1 = t2.col1
```
The analyzed plan should be:
```
Project [col1#0, col2#1, col1#2, col2#3]
+- Join Cross, (col1#0 = col1#2)
   :- SubqueryAlias t1
   :   +- Relation t[col1#0,col2#1] parquet
   +- SubqueryAlias t2
       +- Relation t[col1#2,col2#3] parquet
```
and not:
```
Project [col1#0, col2#1, col1#0, col2#1]
+- Join Cross, (col1#0 = col1#0)
   :- SubqueryAlias t1
   :   +- Relation t[col1#0,col2#1] parquet
   +- SubqueryAlias t2
       +- Relation t[col1#0,col2#1] parquet
```
Because in the latter case the join condition is always true.
For DataFrame programs we need the full power of ExpressionIdAssigner, and old ID -> new ID mapping comes in handy, because DataFrame programs pass _partially_ resolved plans to the Resolver, which may consist of duplicate subtrees, and thus will have already assigned expression IDs. These already resolved duplicate subtrees with assigned IDs will conflict. Hence, we need to reallocate all the leaf node outputs _and_ remap old IDs to the new ones. Also, DataFrame programs may introduce the same Aliases in different parts of the query plan, so we just reallocate all the Aliases.
For example, consider this DataFrame program:
```
spark.range(0, 10).select($"id").write.format("parquet").saveAsTable("t")
val alias = ($"id" + 1).as("id")
spark.table("t").select(alias).select(alias)
```
The analyzed plan should be:
```
Project [(id#6L + cast(1 as bigint)) AS id#13L]
+- Project [(id#4L + cast(1 as bigint)) AS id#6L]
   +- SubqueryAlias spark_catalog.default.t
      +- Relation spark_catalog.default.t[id#4L] parquet
```
and not:
```
Project [(id#6L + cast(1 as bigint)) AS id#6L]
+- Project [(id#4L + cast(1 as bigint)) AS id#6L]
   +- SubqueryAlias spark_catalog.default.t
      +- Relation spark_catalog.default.t[id#4L] parquet
```
Because the latter case will confuse the Optimizer and the top Project will be eliminated leading to incorrect result.
There's an important caveat here: the leftmost branch of a logical plan tree. In this branch we need to preserve the expression IDs wherever possible because DataFrames may reference each other using their attributes. This also makes sense for performance reasons.
Consider this example:
```
val df1 = spark.range(0, 10).select($"id")
val df2 = spark.range(5, 15).select($"id")
df1.union(df2).filter(df1("id") === 5)
```
In this example df("id") references lower id attribute by expression ID, so union must not reassign expression IDs in df1 (left child). Referencing df2 (right child) is not supported in Spark.
The ExpressionIdAssigner covers both SQL and DataFrame scenarios with single approach and is integrated in the single-pass analysis framework.
The ExpressionIdAssigner is used in the following way:
- When the Resolver traverses the tree downwards prior to starting bottom-up analysis, we build the mappingStack by calling withNewMapping (i.e. mappingStack.push) for every child of a multi-child operator, so we have a separate stack entry (separate mapping) for each branch. This way sibling branches' mappings are isolated from each other and attribute IDs are reused only within the same branch. Initially we push None, because the mapping needs to be initialized later with the correct output of a resolved operator.
- When the bottom-up analysis starts, we assign IDs to all the NamedExpressions which are present in operators starting from the LeafNodes using mapExpression. createMapping is called right after each LeafNode is resolved, and first remapped attributes come from that LeafNode. This is done in Resolver.handleLeafOperator for each logical plan tree branch except the leftmost.
- Once the child branch is resolved, withNewMapping ends by calling mappingStack.pop.
- After the multi-child operator is resolved, we call createMapping to initialize the mapping with attributes _chosen_ (e.g. Union.mergeChildOutputs) by that operator's resolution algorithm and remap _old_ expression IDs to those chosen attributes.
- Continue remapping expressions until we reach the root of the operator tree.
class ExpressionResolutionContext extends AnyRef
The ExpressionResolutionContext is a state that is propagated between the nodes of the expression tree during the bottom-up expression resolution process.
The ExpressionResolutionContext is a state that is propagated between the nodes of the expression tree during the bottom-up expression resolution process. This way we pass the results of ExpressionResolver.resolve call, which are not the resolved child itself, from children to parents.
class ExpressionResolutionValidator extends AnyRef
The ExpressionResolutionValidator performs the validation work on the expression tree for the ResolutionValidator.
The ExpressionResolutionValidator performs the validation work on the expression tree for the ResolutionValidator. These two components work together recursively validating the logical plan. You can find more info in the ResolutionValidator scaladoc.
class ExpressionResolver extends TreeNodeResolver[Expression, Expression] with ProducesUnresolvedSubtree with ResolvesExpressionChildren
The ExpressionResolver is used by the Resolver during the analysis to resolve expressions.
The ExpressionResolver is used by the Resolver during the analysis to resolve expressions.
The functions here generally traverse unresolved Expression nodes recursively, constructing and returning the resolved Expression nodes bottom-up. This is the primary entry point for implementing expression analysis, wherein the resolve method accepts a fully unresolved Expression and returns a fully resolved Expression in response with all data types and attribute reference ID assigned for valid requests. This resolver also takes responsibility to detect any errors in the initial SQL query or DataFrame and return appropriate error messages including precise parse locations wherever possible.
class FunctionResolver extends TreeNodeResolver[UnresolvedFunction, Expression] with ProducesUnresolvedSubtree
A resolver for UnresolvedFunctions that resolves functions to concrete Expressions.
A resolver for UnresolvedFunctions that resolves functions to concrete Expressions. It resolves the children of the function first by calling ExpressionResolver.resolve on them if they are not UnresolvedStars. If the children are UnresolvedStars, it resolves them using ExpressionResolver.resolveStar. Examples are following:
- Function doesn't contain any UnresolvedStar:
```
SELECT ARRAY(col1) FROM VALUES (1);
```
it is resolved only using ExpressionResolver.resolve.
- Function contains UnresolvedStar:
```
SELECT ARRAY(*) FROM VALUES (1);
```
it is resolved using ExpressionResolver.resolveStar.
After resolving the function with FunctionResolution.resolveFunction, performs further resolution in following cases:
- result of FunctionResolution.resolveFunction is ExpressionWithRandomSeed in which case it is necessary to initialize the random seed.
- result of FunctionResolution.resolveFunction is InheritAnalysisRules, in which case it is necessary to resolve its replacement expression.
- result of FunctionResolution.resolveFunction is AggregateExpression or BinaryArithmetic, in which case it is necessary to perform further resolution and checks on its children.
Finally apply type coercion to the result of previous step and in case that the resulting expression is TimeZoneAwareExpression, apply timezone.
class HybridAnalyzer extends SQLConfHelper
The HybridAnalyzer routes the unresolved logical plan between the legacy Analyzer and a single-pass Analyzer when the query that we are processing is being run from unit tests depending on the testing flags set and the structure of this unresolved logical plan:
The HybridAnalyzer routes the unresolved logical plan between the legacy Analyzer and a single-pass Analyzer when the query that we are processing is being run from unit tests depending on the testing flags set and the structure of this unresolved logical plan:
- If the "spark.sql.analyzer.singlePassResolver.soloRunEnabled" is "true", the HybridAnalyzer will unconditionally run the single-pass Analyzer, which would usually result in some unexpected behavior and failures. This flag is used only for development.
- If the "spark.sql.analyzer.singlePassResolver.dualRunEnabled" is "true", the HybridAnalyzer will invoke the legacy analyzer and optionally _also_ the fixed-point one depending on the structure of the unresolved plan. This decision is based on which features are supported by the single-pass Analyzer, and the checking is implemented in the ResolverGuard. After that we validate the results using the following logic:
  - If the fixed-point Analyzer fails and the single-pass one succeeds, we throw an appropriate exception (please check the QueryCompilationErrors.fixedPointFailedSinglePassSucceeded method)
  - If both the fixed-point and the single-pass Analyzers failed, we throw the exception from the fixed-point Analyzer.
  - If the single-pass Analyzer failed, we throw an exception from its failure.
  - If both the fixed-point and the single-pass Analyzers succeeded, we compare the logical plans and output schemas, and return the resolved plan from the fixed-point Analyzer.
- Otherwise we run the legacy analyzer.
class LateralColumnAliasProhibitedRegistry extends LateralColumnAliasRegistry
Dummy implementation of LateralColumnAliasRegistry used when SQLConf.LATERAL_COLUMN_ALIAS_IMPLICIT_ENABLED is disabled.
Dummy implementation of LateralColumnAliasRegistry used when SQLConf.LATERAL_COLUMN_ALIAS_IMPLICIT_ENABLED is disabled. Getter methods throw an exception as they should never be called on a dummy implementation. Non-getter methods must remain idempotent.
abstract class LateralColumnAliasRegistry extends AnyRef
Base class for lateral column alias registry.
Base class for lateral column alias registry. This class is extended by 2 implementations:
1. LateralColumnAliasRegistryImpl - When SQLConf.LATERAL_COLUMN_ALIAS_IMPLICIT_ENABLED is enabled, this class implements logic for LCA resolution. 2. LateralColumnAliasProhibitedRegistry - Dummy class whose methods throw exceptions when LCA resolution is disabled by SQLConf.LATERAL_COLUMN_ALIAS_IMPLICIT_ENABLED.
class LateralColumnAliasRegistryImpl extends LateralColumnAliasRegistry
LateralColumnAliasRegistryImpl is a utility class that contains structures required for lateral column alias resolution.
LateralColumnAliasRegistryImpl is a utility class that contains structures required for lateral column alias resolution. Here we store:
- currentAttributeDependencyLevelStack - Current attribute dependency level in the scope. Dependency level is defined as a maximum dependency in that attribute's expression tree. For example, in a query like:
```
SELECT a, b, a + b AS c, a + c AS d
```
Dependency levels will be as follows: level 0: a, b level 1: c level 2: d
We add a new entry to the stack for each new Alias resolution. This is needed because we can have nesting Aliases in the plan, that do not belong to the same LCA scope. For example, in the following query:
```
SELECT STRUCT('alpha' AS A, 'beta' AS B) ST
```
ST, A and B would be aliases in the same expression tree, but they do not belong in the same LCA scope.
- availableAttributes - All attributes that can be laterally referenced. This map is indexed by name, but contains a list of attributes with the same name. This is because it is possible to have multiple attributes with the same name in the scope, but they can't be laterally referenced. Handling ambiguous references is done in the getAttribute method. For the following query:
```
SELECT 0 AS a, 1 AS b, 2 AS c, b AS d, a AS e, d AS f, a AS g, g AS h, h AS i
```
availableAttributes will be: {a, b, c, d, e, f, g, h, i}
- referencedAliases - Aliases that have been laterally referenced. For the given query example, referencedAliases will be: {a, b, d, g, h}
- aliasDependencyLevels - Dependency levels of all aliases, indexed by dependency level. For the given query example, dependency levels will be as follows:
level 0: a, b, c level 1: d, e, g level 2: f, h level 3: i
class LimitExpressionResolver extends TreeNodeResolver[Expression, Expression]
The LimitExpressionResolver is a resolver that resolves a LocalLimit or GlobalLimit expression and performs all the necessary validation.
type LogicalPlanResolver = TreeNodeResolver[LogicalPlan, LogicalPlan]
class MetadataResolver extends RelationMetadataProvider with DelegatesResolutionToExtensions
The MetadataResolver performs relation metadata resolution based on the unresolved plan at the start of the analysis phase.
The MetadataResolver performs relation metadata resolution based on the unresolved plan at the start of the analysis phase. Usually it does RPC calls to some table catalog and to table metadata itself.
RelationsWithResolvedMetadata is a map from relation ID to the relations with resolved metadata. It's produced by resolve and is used later in Resolver to replace UnresolvedRelations.
This object is one-shot per SQL query or DataFrame program resolution.

class NameScope extends SQLConfHelper

The NameScope is used to control the resolution of names (table, column, alias identifiers).

The NameScope is used to control the resolution of names (table, column, alias identifiers). It's a part of the Resolver's state, and is used to manage the output of SQL query/DataFrame program operators.

The NameScope output is immutable. If it's necessary to update the output, NameScopeStack methods are used (overwriteTop or withNewScope). The NameScope is always used through the NameScopeStack.

The resolution of identifiers is case-insensitive.

Name resolution priority is as follows:

Resolution of local references:
- column reference
- struct field or map key reference 2. Resolution of lateral column aliases (if enabled).

For example, in a query like:

SELECT 1 AS col1, col1 FROM VALUES (2)

Because column resolution has a higher priority than LCA resolution, the result will be [1, 2] and not [1, 1].

Approximate tree of NameScope manipulations is shown in the following example:

CREATE TABLE IF NOT EXISTS t1 (col1 INT, col2 INT, col3 STRING);

SELECT
  col1, col2 as alias1
FROM
  (SELECT * FROM VALUES (1, 2))
  UNION
  (SELECT t2.col1, t2.col2 FROM (SELECT col1, col2 FROM t1) AS t2)
;

unionAttributes = withNewScope {
  lhsOutput = withNewScope {
    expandedStar = withNewScope {
      scope.overwriteTop(localRelation.output)
      scope.expandStar(star)
    }
    scope.overwriteTop(expandedStar)
    scope.output
  }
  rhsOutput = withNewScope {
    subqueryAttributes = withNewScope {
      scope.overwriteTop(t1.output)
      scope.overwriteTop(prependQualifier(scope.output, "t2"))
      [scope.matchMultiPartName("t2", "col1"), scope.matchMultiPartName("t2", "col2")]
    }
    scope.overwriteTop(subqueryAttributes)
    scope.output
  }
  scope.overwriteTop(coerce(lhsOutput, rhsOutput))
  [scope.matchMultiPartName("col1"), alias(scope.matchMultiPartName("col2"), "alias1")]
}
scope.overwriteTop(unionAttributes)

class NameScopeStack extends SQLConfHelper
The NameScopeStack is a stack of NameScopes managed by the Resolver.
The NameScopeStack is a stack of NameScopes managed by the Resolver. Usually a top scope is used for name resolution, but in case of correlated subqueries we can lookup names in the parent scopes. Low-level scope creation is managed internally, and only high-level api like withNewScope is available to the resolvers. Freshly-created NameScopeStack contains an empty root NameScope, which in the context of Resolver corresponds to the query output.
case class NameTarget(candidates: Seq[Expression], aliasName: Option[String] = None, lateralAttributeReference: Option[Attribute] = None, output: Seq[Attribute] = Seq.empty) extends Product with Serializable
NameTarget is a result of a multipart name resolution of the NameScope.resolveMultipartName.
NameTarget is a result of a multipart name resolution of the NameScope.resolveMultipartName.
Attribute resolution:
```
-- [[NameTarget]] with a single candidate `col1`. `aliasName` is be `None` in this case because
-- the name is not a field/value/item of some recursive type.
SELECT col1 FROM VALUES (1);
```
Attribute resolution ambiguity:
```
-- [[NameTarget]] with candidates `col1`, `col1`. [[pickCandidate]] will throw
-- `AMBIGUOUS_REFERENCE`.
SELECT col1 FROM VALUES (1) t1, VALUES (2) t2;
```
Struct field resolution:
```
-- [[NameTarget]] with a single candidate `GetStructField(col1, "field1")`. `aliasName` is
-- `Some("col1")`, since here we extract a field of a struct.
SELECT col1.field1 FROM VALUES (named_struct('field1', 1), 3);
```
candidates
A list of candidates that are possible matches for a given name.
aliasName
If the candidates size is 1 and it's type is ExtractValue (which means that it's a field/value/item from a recursive type), then the aliasName should be the name with which the candidate needs to be aliased. Otherwise, aliasName is None.
lateralAttributeReference
If the candidate is laterally referencing another column this field is populated with that column's attribute.
output
output of a NameScope that produced this NameTarget. Used to provide suggestions for thrown errors.
class PlanLogger extends Logging
PlanLogger is used by the Resolver to log intermediate resolution results.
class PredicateResolver extends TreeNodeResolver[Predicate, Expression] with ResolvesExpressionChildren
Resolver class for resolving all Predicate expressions.
Resolver class for resolving all Predicate expressions. Recursively resolves all children and applies selected type coercions to the expression.
trait ProducesUnresolvedSubtree extends ResolvesExpressionChildren
A mixin trait for expression resolvers that as part of their resolution, replace single node with a subtree of nodes.
A mixin trait for expression resolvers that as part of their resolution, replace single node with a subtree of nodes. This step is necessary because the underlying legacy code that is being called produces partially-unresolved subtrees. In order to resolve the subtree a callback resolver is called recursively. This callback must ensure that no node is resolved twice in order to not break the single-pass invariant. This is done by tagging the limits of this traversal with ExpressionResolver.SINGLE_PASS_SUBTREE_BOUNDARY tag. This tag is applied to the original expression's children, which are guaranteed to be resolved at the time of given expression's resolution. When callback resolver encounters the node that is tagged, it should return identity instead of trying to resolve it.
class ProhibitedResolver extends LogicalPlanResolver
This is a dummy LogicalPlanResolver whose resolve is not implemented and throws SparkException.
This is a dummy LogicalPlanResolver whose resolve is not implemented and throws SparkException.
It's used by the MetadataResolver to pass it as an argument to tryDelegateResolutionToExtensions, because unresolved subtree resolution doesn't make sense during metadata resolution traversal.
class ProjectResolver extends TreeNodeResolver[Project, LogicalPlan]
Resolves initially unresolved Project operator to either a resolved Project or Aggregate node, based on whether there are aggregate expressions in the project list.
Resolves initially unresolved Project operator to either a resolved Project or Aggregate node, based on whether there are aggregate expressions in the project list. When LateralColumnAlias resolution is enabled, replaces the output operator with an appropriate operator structure using information from the scope. Detailed explanation can be found in buildProjectWithResolvedLCAs method.
case class RelationId(multipartIdentifier: Seq[String], options: CaseInsensitiveStringMap = CaseInsensitiveStringMap.empty, isStreaming: Boolean = false) extends Product with Serializable
The RelationId is a unique identifier for a relation.
The RelationId is a unique identifier for a relation. It is used to lookup the relations which were processed by the MetadataResolver to substitute the unresolved relations in single pass during the analysis phase.
trait RelationMetadataProvider extends LookupCatalog
RelationMetadataProvider provides relations with resolved metadata based on the corresponding UnresolvedRelations.
RelationMetadataProvider provides relations with resolved metadata based on the corresponding UnresolvedRelations. It is used by Resolver to replace UnresolvedRelation with a specific LogicalPlan with resolved metadata, e.g. with UnresolvedCatalogRelation or View.
class ResolutionValidator extends AnyRef
The ResolutionValidator performs the validation work after the logical plan tree is resolved by the Resolver.
The ResolutionValidator performs the validation work after the logical plan tree is resolved by the Resolver. Each resolve* method in the Resolver must have its validate* counterpart in the ResolutionValidator. The validation code asserts the conditions that must never be false no matter which SQL query or DataFrame program was provided. The validation approach is single-pass, post-order, complementary to the resolution process.
case class ResolvedProjectList(expressions: Seq[NamedExpression], hasAggregateExpressions: Boolean, hasAttributes: Boolean, hasLateralColumnAlias: Boolean) extends Product with Serializable
Structure used to return results of the resolved project list.
Structure used to return results of the resolved project list.
- expressions: The resolved expressions. It is resolved using the resolveExpressionTreeInOperator.
- hasAggregateExpressions: True if the resolved project list contains any aggregate expressions.
- hasAttributes: True if the resolved project list contains any attributes that are not under an aggregate expression.
- hasLateralColumnAlias: True if the resolved project list contains any lateral column aliases.
class Resolver extends LogicalPlanResolver with ResolvesOperatorChildren with DelegatesResolutionToExtensions
The Resolver implements a single-pass bottom-up analysis algorithm in the Catalyst.
The Resolver implements a single-pass bottom-up analysis algorithm in the Catalyst.
The functions here generally traverse the LogicalPlan nodes recursively, constructing and returning the resolved LogicalPlan nodes bottom-up. This is the primary entry point for implementing SQL and DataFrame plan analysis, wherein the resolve method accepts a fully unresolved LogicalPlan and returns a fully resolved LogicalPlan in response with all data types and attribute reference ID assigned for valid requests. This resolver also takes responsibility to detect any errors in the initial SQL query or DataFrame and return appropriate error messages including precise parse locations wherever possible.
The Resolver is a one-shot object per each SQL/DataFrame logical plan, the calling code must re-create it for every new analysis run.
trait ResolverExtension extends AnyRef
The ResolverExtension is a main interface for single-pass analysis extensions in Catalyst.
The ResolverExtension is a main interface for single-pass analysis extensions in Catalyst. External code that needs specific node types to be resolved has to implement this trait and inject the implementation into the Analyzer.singlePassResolverExtensions.
class ResolverGuard extends SQLConfHelper
ResolverGuard is a class that checks if the operator that is yet to be analyzed only consists of operators and expressions that are currently supported by the single-pass analyzer.
ResolverGuard is a class that checks if the operator that is yet to be analyzed only consists of operators and expressions that are currently supported by the single-pass analyzer.
This is a one-shot object and should not be reused after apply call.
class ResolverRunner extends SQLConfHelper
Wrapper class for Resolver and single-pass resolution.
Wrapper class for Resolver and single-pass resolution. This class encapsulates single-pass resolution and post-processing of resolved plan. This post-processing is necessary in order to either fully resolve the plan or stay compatible with the fixed-point analyzer.
trait ResolvesExpressionChildren extends AnyRef
trait ResolvesOperatorChildren extends AnyRef
A mixin trait for all operator resolvers that need to resolve their children.
class TimeAddResolver extends TreeNodeResolver[TimeAdd, Expression] with ResolvesExpressionChildren
Helper resolver for TimeAdd which is produced by resolving BinaryArithmetic nodes.
class TimezoneAwareExpressionResolver extends TreeNodeResolver[TimeZoneAwareExpression, Expression] with ResolvesExpressionChildren
Resolves TimeZoneAwareExpressions by applying the session's local timezone.
Resolves TimeZoneAwareExpressions by applying the session's local timezone.
This class is responsible for resolving TimeZoneAwareExpressions by first resolving their children and then applying the session's local timezone. Additionally, ensures that any tags from the original expression are preserved during the resolution process.
trait TreeNodeResolver[UnresolvedNode <: TreeNode[_], ResolvedNode <: TreeNode[_]] extends SQLConfHelper with QueryErrorsBase
Base class for TreeNode resolvers.
Base class for TreeNode resolvers. All resolvers should extend this class with specific UnresolvedNode and ResolvedNode types.
class TypeCoercionResolver extends TreeNodeResolver[Expression, Expression]
TypeCoercionResolver is used by other resolvers to uniformly apply type coercions to all expressions.
class UnaryMinusResolver extends TreeNodeResolver[UnaryMinus, Expression] with ResolvesExpressionChildren
Resolver for UnaryMinus.
Resolver for UnaryMinus. Resolves children and applies type coercion to target node.
class UnionResolver extends TreeNodeResolver[Union, Union]
The UnionResolver performs Union operator resolution.
The UnionResolver performs Union operator resolution. This operator has 2+ children. Resolution involves checking and normalizing child output attributes (data types and nullability).
case class ViewResolutionContext(nestedViewDepth: Int, maxNestedViewDepth: Int) extends Product with Serializable
The ViewResolutionContext consists of data, which is specific to the specific view plan resolution.
The ViewResolutionContext consists of data, which is specific to the specific view plan resolution. This data is also propagated to the subviews.
nestedViewDepth
Current nested view depth. Cannot exceed the maxNestedViewDepth.
maxNestedViewDepth
Maximum allowed nested view depth. Configured in the upper context based on SQLConf.MAX_NESTED_VIEW_DEPTH.
class ViewResolver extends TreeNodeResolver[View, View]
The ViewResolver resolves view plans that were already reconstructed by SessionCatalog from the view text and view metadata (schema, configs).

Value Members

object AggregateExpressionResolver
object AnalyzerBridgeState extends Serializable
object BinaryArithmeticResolver
object ConditionalExpressionResolver
object CreateNamedStructResolver
object ExplicitlyUnsupportedResolverFeature extends Serializable
This object contains all the metadata on explicitly unsupported resolver features.
object ExpressionIdAssigner
object ExpressionResolver
object PredicateResolver
object Resolver
object ResolverGuard
object TimeAddResolver
object TypeCoercionResolver
object UnaryMinusResolver

Packages

resolver

package resolver

Type Members

Value Members

Inherited from AnyRef

Inherited from Any

Ungrouped

Packages

resolver

package resolver

Type Members

Value Members

Inherited from AnyRef

Inherited from Any

Ungrouped

resolver