The active config object within the current scope. See SQLConf.get for more information.

Deduplicates missing expressions by ExprId.

Insert the missing expressions in the output list of the operator. Recursively call expandOperatorsOutputList to expand the output lists of Projects and Aggregates below the current one. In order to stay compatible with fixed-point, missing expressions are inserted after the original output list, but before any qualified access only columns that have been added as part of resolution from hidden output.

Only AttributeReferences are propagated recursively in expandOperatorsOutputList. Aliases are meant to be inserted in the topmost operator. For example, SortResolver may push down aliased aggregate and grouping expression trees to the immediate Aggregate below, but it does not make sense to push them down further:

-- The MAX(v2.col1) aggregate has to be pushed down to [[Aggregate]], but not to [[Project]]
-- below it.
SELECT COUNT(col1) FROM v1 NATURAL JOIN v2 GROUP BY col1 ORDER BY MAX(v2.col1);

->

Project [count(col1)#23L]
+- Sort [max(col1)#25 ASC NULLS FIRST], true
  +- Aggregate [col1#21], [count(col1#21) AS count(col1)#23L, max(col1#20) AS max(col1)#25]
    +- Project [col1#21, col1#20]
      ...

Resolve Sort operator.

1. Resolve Sort.child and set NameScope.ordinalReplacementExpressions for SortOrder resolution. 2. Resolve order expressions using ExpressionResolver.resolveExpressionTreeInOperator. 3. In case order expressions contain only one element, UnresolvedAttribute(ALL), which can't be resolved from current (nor from hidden output) - this is done using ResolveAsAllKeyword, skip previous step and resolve it as an ALL keyword (by copying child's output and transforming it into attributes). 4. In case there are attributes referenced in the order expressions are resolved using the hidden output (or in case we order by AggregateExpressions which are not present in Aggregate.aggregateExpressions) update the output of child operator and add a Project as a parent of Sort with original Project's output (this is done by ResolvesNameByHiddenOutput). Query:

SELECT col1 FROM VALUES(1, 2) WHERE col2 > 2 ORDER BY col2;

Has the following unresolved plan:

'Sort ['col2 ASC NULLS FIRST], true +- 'Project ['col1] +- 'Filter ('col2 > 2) +- LocalRelation [col1#92225, col2#92226]

Because col2 from the Sort node is resolved using the hidden output, add it to the Project.projectList and add a Project with original project list as a top node:

Project [col1] +- Sort [col2 ASC NULLS FIRST], true +- Project [col1, col2] +- Filter (col2 > 2) +- LocalRelation [col1, col2]

Another example with ordering by AggregateExpression:

SELECT col1 FROM VALUES (1, 2) GROUP BY col1, col2 + 1 ORDER BY SUM(col1), col2 + 1;

Has the following unresolved plan:

'Sort ['SUM('col1) ASC NULLS FIRST, ('col2 + 1) ASC NULLS FIRST], true +- 'Aggregate ['col1, ('col2 + 1)], ['col1] +- LocalRelation [col1, col2]

Because neither SUM(col1) nor col2 + 1 from the Sort node are present in the Aggregate.aggregateExpressions, add them to it and add a Project with original project list as a top node (SUM(col2) is extracted in the AggregateExpressionResolver whereas col2 + 1 is extracted using extractReferencedGroupingAndAggregateExpressions helper method):

Project [col1] +- Sort [sum(col1)#... ASC NULLS FIRST, (col2 + 1)#... ASC NULLS FIRST], true +- Aggregate [col1, (col2 + 1)], [col1, sum(col1) AS sum(col1)#..., (col2 + 1) AS (col2 + 1)#...] +- LocalRelation [col1, col2]

If missingExpressions is not empty, output of an operator has been changed by insertMissingExpressions. Therefore, we need to restore the original output, by placing a Project on top of an original node, with original's node output. Additionally, we append all qualified access only columns from hidden output that were inserted as missing attributes, because they may be needed in upper operators (if not, they will be pruned away in PruneMetadataColumns). Other hidden attributes are thrown away, because we cannot reference them from the new Project (they are not outputted from below).

If SQLConf.SINGLE_PASS_RESOLVER_PREVENT_USING_ALIASES_FROM_NON_DIRECT_CHILDREN is set to true, we need to overwrite the current scope and clear aggregateListAliases and baseAggregate. This is needed in order to prevent later replacement of Sort/Having expressions using semantically equal aliased expressions from non-direct children. For example, in the following query:

SELECT col1 AS a FROM VALUES(1,2) GROUP BY col1, col2 HAVING col2 > 1 ORDER BY col1;

With flag set to false, analyzed plan will be:

Sort [a#3 ASC NULLS FIRST], true +- Project [a#3] +- Filter (col2#2 > 1) +- Aggregate [col1#1, col2#2], [col1#1 AS a#3, col2#2, col1#1] +- LocalRelation [col1#1, col2#2]

Instead of using missing attribute col1#1 we can use its alias a#3 in the Sort and avoid adding an extra projection. This is because all of Sort, Project, Filter and Aggregate belong to the same NameScope since Project was artificially inserted.

However, fixed-point can't handle this case properly and produces the following plan:

Project [a#3] +- Sort [col1#1 ASC NULLS FIRST], true +- Project [a#3, col1#1] +- Filter (col2#2 > 1) +- Aggregate [col1#1, col2#2], [col1#1 AS a#3, col2#2, col1#1] +- LocalRelation [col1#1, col2#2]

Therefore, we need to match this behavior of fixed-point in single-pass in order to avoid logical plan mismatches.

When resolving lateral column references in Aggregate below Sort or HAVING operators, fixed-point first resolves SortOrder expressions and HAVING conditions using TempResolvedColumn and only after that resolves lateral column references. For example, consider the following query:

SELECT avg(col1) AS a, a AS b FROM VALUES(1,2,3) GROUP BY col2 ORDER BY max(col3)

Fixed-point plan before resolving SortOrder:

Sort [max(tempresolvedcolumn(col3#5, col3, false)) ASC NULLS FIRST], true +- Aggregate [col2#4], [avg(col1#3) AS a#6, lateralAliasReference(a) AS b#7] +- LocalRelation [col1#3, col2#4, col3#5]

After resolving TempResolvedColumn:

Project [a#6, b#7] +- Sort [max(col3)#10 ASC NULLS FIRST], true +- Aggregate [col2#4], [avg(col1#3) AS a#6, lca(a) AS b#7, max(col3#5) AS max(col3)#10] +- LocalRelation [col1#3, col2#4, col3#5]

In the above case fixed-point first resolves SortOrder to max(col3)#10 and only then resolves LCAs. However, while resolving LCAs in Aggregate, fixed-point first constructs a base Aggregate by pushing down all aggregate expressions with new aliases. It then places a Project on top reinstating the original alias on top of a newly created one, in order to still match the attribute reference from SortOrder:

Project [a#6, b#7] +- Sort [max(col3)#10 ASC NULLS FIRST], true +- Project [avg(col1)#11 AS a#6, lca(a) AS b#7, max(col3)#12 AS max(col3)#10] +- Aggregate [col2#4], [avg(col1#3) AS avg(col1)#11, max(col3#5) AS max(col3)#12] +- LocalRelation [col1#3, col2#4, col3#5]

In the example above, max(col3#5) gets pushed down and aliased as max(col3)#12, even though max(col3)#10 attribute reference already exists. Because of that max(col3)#12 needs to be remapped back to max(col3)#10.

However, in single-pass analyzer, we will first resolve all lateral column references before starting the resolution of SortOrder resulting in the following plan:

Project [a#6, b#7] +- Sort [max(col3)#16 ASC NULLS FIRST], true +- Project [a#6, a#6 AS b#7, max(col3)#16] +- Project [avg(col1)#14, avg(col1)#14 AS a#6, max(col3)#16] +- Aggregate [col2#4], [avg(col1#3) AS avg(col1)#14, max(col3#5) AS max(col3)#16] +- LocalRelation [col1#3, col2#4, col3#5]

In the above case, rewriting max(col3)#16 with an Alias is not necessary from correctness perspective, but we need to do it in order to stay compatible with fixed-point analyzer. Because fixed-point only regenerates aliases from original aggregate list, in single-pass we need to handle the following:

1. all aliases from top-level Project (because they originate from the unresolved aggregate list); 2. all references to aliases from the base aggregate (because they are became attribute references during LCA resolution);

This same issue also applies to HAVING resolution.

When resolving Sort or Having on top of an Aggregate that has lateral column references, aggregate and grouping expressions might not be correctly replaced in SortOrder and HAVING condition, because of Project nodes created when resolving lateral column references. Because of that, we need to additionally try and replace SortOrder expressions and HAVING conditions that don't appear in the child Project, but the aliases of semantically equivalent expressions do. In case both the attribute and its alias exist in the output, don't replace the attribute in SortOrder / HAVING condition, because there is no missing input in that case. For example, consider the following query:

SELECT col1 AS a, a FROM VALUES(1) GROUP BY col1 ORDER BY col1

After resolving lateral column references and partially resolving SortOrder expression, we get the following plan:

!Sort [col1#3 ASC NULLS FIRST], true +- Project [a#4, a#4] +- Project [col1#3, col1#3 AS a#4] +- Aggregate [col1#3], [col1#3] +- LocalRelation [col1#3]

In the above plan, Sort has a missing input col1#3. Because of LCA resolution this attribute is pushed down into the Project stack and aliased as a#4. Instead of using col1#3 we can reference its semantically equivalent alias a#4 in the SortOrder. The resolved plan looks like:

Sort [a#4 ASC NULLS FIRST], true +- Project [a#4, a#4] +- Project [col1#3, col1#3 AS a#4] +- Aggregate [col1#3], [col1#3] +- LocalRelation [col1#3]

Because we used a#4 alias instead of col1#3, we do not need to insert col1#3 to the child Project as a missing expression. Therefore, missingExpressions need to be updated in order not to insert unnecessary attributes in ResolvesNameByHiddenOutput.insertMissingExpressions

However, for a query like:

SELECT col1, col1 AS a FROM VALUES(1) GROUP BY col1 ORDER BY col1

The resolved plan will be:

Sort [col1#4 ASC NULLS FIRST], true +- Aggregate [col1#4], [col1#4, col1#4 AS a#5] +- LocalRelation [col1#4]

In the above example, we do not replace col1#4 with a#5 because col1#4 is present in the output.

Sets all SQL configurations specified in pairs, calls f, and then restores all SQL configurations.