ResolvesNameByHiddenOutput is used by resolvers for operators that are able to resolve attributes in its expression tree from hidden output or that can reference expressions not present in child's output. Update child operator's output list and place a Project node on top of original operator node with the original output of an operator's child.

For example, in a following query:

SELECT
 t1.key
FROM
 t1 FULL OUTER JOIN t2 USING (key)
WHERE
 t1.key NOT LIKE 'bb.%';

Plan without adding missing attributes would be:

+- Project [key#1]
   +- Filter NOT key#1 LIKE bb.%
      +- Project [coalesce(key#1, key#2) AS key#3, __key#1__, __key#2__]
         +- Join FullOuter, (key#1 = key#2)
            :- SubqueryAlias t1
            :  +- Relation t1[key#1]
            +- SubqueryAlias t2
               +- Relation t2[key#2]

NOTE: #key1 and key#2 at the end of inner Project are metadata columns from the full outer join. Even though they are present in the Project in single-pass, fixed-point adds these columns after resolving missing input, so duplication of some columns is possible. In order to stay fully compatible between single-pass and fixed-point, we add both missing attributes and these metadata columns. We mimic fixed-point behavior by putting metadata columns in NameScope.hiddenOutput instead of NameScope.output.

In the plan above, Filter requires key#1 in its condition, but key#1 is not available in the below Project's output, even though key#1 is available in Join's hidden output. Because of that, we need to place key#1 in the project list, after original project list expressions, but before metadata columns (to remain compatible with fixed-point). In order to preserve initial output of Filter, we place a Project node on top of this Filter, whose project list is the original output of the Project below Filter (in this case - key#3 and metadata columns key#1 and key#2).

Therefore, the plan becomes:

+- Project [key#1]
   +- Project [key#3, key#1, key#2]
      +- Filter NOT key#1 LIKE bb.%
         +- Project [coalesce(key#1, key#2) AS key#3, key#1, key#1, key#2]
            +- Join FullOuter, (key#1 = key#2)
               :- SubqueryAlias t1
               :  +- Relation t1[key#1]
               +- SubqueryAlias t2
                  +- Relation t2[key#2]

Query below exhibits similar behavior when Sort operator resolves an attribute using hidden output:

SELECT col1 FROM VALUES (1, 2) ORDER BY col2;

Unresolved plan would be:

Sort [col2 ASC NULLS FIRST], true
  +- Project [col1]
    +- LocalRelation [col1, col2]

As it can be seen, attribute col2 used in Sort can't be resolved using the Project output (which is [col1]), so it has to be resolved using the hidden output (which is propagated from LocalRelation and is [col1, col2]). As it's been shown in the previous example, col2 has to be added to Project list and a Project with original output of the Project below Sort is added as a top node. Because of that, analyzed plan is:

Project [col1]
  +- Sort [col2 ASC NULLS FIRST], true
    +- Project [col1, col2]
      +- LocalRelation [col1, col2]

Another example is when Sort order expression is an AggregateExpression which is not present in the Aggregate.aggregateExpressions:

SELECT col1 FROM VALUES (1) GROUP BY col1 ORDER BY sum(col1);

In this example sum(col1) should be added to child's output and a Project node should be added on top of the Sort node to preserve the original output of the Aggregate node:

Project [col1] +- Sort [sum(col1)#... ASC NULLS FIRST], true +- Aggregate [col1], [col1, sum(col1) AS sum(col1)#...] +- LocalRelation [col1]

In case of Dataframe programs we can have multiple Sort operators nested inside each other. For example:

df.select("col1").orderBy("col2").orderBy("col1").orderBy("col2")

Unresolved plan would be:

Sort [col2 ASC NULLS FIRST], true
  +- Sort [col1 ASC NULLS FIRST], true
    +- Project [col1]
      +- Sort [col2 ASC NULLS FIRST], true
        +- Project [col1, col2]
          +- Project [col1, col2, col3]
            +- LocalRelation [col1, col2, col3]

As it can be seen, col2 (Sort order expression) needs to be resolved using the hidden output. Because of that it must be added to all the Projects and Aggregates below the Sort operator. Resolved plan would be:

Project [col1]
  +- Sort [col2 ASC NULLS FIRST], true
    +- Sort [col1 ASC NULLS FIRST], true
      +- Project [col1, col2]
        +- Sort [col2 ASC NULLS FIRST], true
          +- Project [col1, col2]
            +- Project [col1, col2, col3]
              +- LocalRelation [col1, col2, col3]

In the plan you can see that col2 is added to the lower Project.projectList.