Defines a sequence of rule batches, to be overridden by the implementation.

Checks for errors in a SELECT clause, such as a trailing comma or an empty select list.

The active config object within the current scope. See SQLConf.get for more information.

Returns the current catalog set.

Once batches that are excluded in the idempotence checker

Executes the batches of rules defined by the subclass. The batches are executed serially using the defined execution strategy. Within each batch, rules are also executed serially.

Executes the batches of rules defined by the subclass, and also tracks timing info for each rule using the provided tracker.

Override to provide additional checks for correct analysis. These rules will be evaluated after our built-in check rules.

Fails the analysis at the point where a specific tree node was parsed using a provided error class and message parameters.

If the plan cannot be resolved within maxIterations, analyzer will throw exception to inform user to increase the value of SQLConf.ANALYZER_MAX_ITERATIONS.

Look up variable by nameParts. If in SQL Script, first check local variables, unless in EXECUTE IMMEDIATE (EXECUTE IMMEDIATE generated query cannot access local variables). if not found fall back to session variables.

Name for this rule executor, automatically inferred based on class name.

Override to provide rules to do post-hoc resolution. Note that these rules will be executed in an individual batch. This batch is to run right after the normal resolution batch and execute its rules in one pass.

Replace all attributes, that reference an alias, with the aliased expression

Replace all attributes, that reference an alias, with the aliased expression, but keep the name of the outermost attribute.

The last resort to resolve columns. Currently it does two things:

• Try to resolve column names as outer references
• Try to resolve column names as SQL variable

Resolves UnresolvedAttribute, GetColumnByOrdinal and extract value expressions(s) by the input plan's children output attributes.

Resolves UnresolvedAttribute, GetColumnByOrdinal and extract value expressions(s) by the input plan's output attributes. In order to resolve the nested fields correctly, this function makes use of throws parameter to control when to raise an AnalysisException.

Example : SELECT * FROM t ORDER BY a.b

In the above example, after a is resolved to a struct-type column, we may fail to resolve b if there is no such nested field named "b". We should not fail and wait for other rules to resolve it if possible.

This method tries to resolve expressions and find missing attributes recursively. Specifically, when the expressions used in Sort or Filter contain unresolved attributes or resolved attributes which are missing from child output. This method tries to find the missing attributes and add them into the projection.

Extensions used for early resolution of the single-pass analyzer.

See ResolverExtension for more info.

Extensions for the single-pass analyzer.

See ResolverExtension for more info.

Defines a validate function that validates the plan changes after the execution of each rule, to make sure these rules make valid changes to the plan. For example, we can check whether a plan is still resolved after each rule in Optimizer, so that we can catch rules that turn the plan into unresolved.

Defines a validate function that validates the plan changes after the execution of each rule, to make sure these rules make valid changes to the plan. Since this is enabled by default, this should only consist of very lightweight checks.

Sets all SQL configurations specified in pairs, calls f, and then restores all SQL configurations.

Adds metadata columns to output for child relations when nodes are missing resolved attributes.

References to metadata columns are resolved using columns from LogicalPlan.metadataOutput, but the relation's output does not include the metadata columns until the relation is replaced. Unless this rule adds metadata to the relation's output, the analyzer will detect that nothing produces the columns.

This rule only adds metadata columns when a node is resolved but is missing input from its children. This ensures that metadata columns are not added to the plan unless they are used. By checking only resolved nodes, this ensures that * expansion is already done so that metadata columns are not accidentally selected by *. This rule resolves operators downwards to avoid projecting away metadata columns prematurely.

Extract legacy table identifier from a multi-part identifier.

For legacy support only. Please use CatalogAndIdentifier instead on DSv2 code paths.

Extract catalog and identifier from a multi-part name with the current catalog if needed. Catalog name takes precedence over identifier, but for a single-part name, identifier takes precedence over catalog name.

Note that, this pattern is used to look up permanent catalog objects like table, view, function, etc. If you need to look up temp objects like temp view, please do it separately before calling this pattern, as temp objects don't belong to any catalog.

Extract catalog and namespace from a multi-part name with the current catalog if needed. Catalog name takes precedence over namespaces.

Extracts Generator from the projectList of a Project operator and creates Generate operator under Project.

This rule will throw AnalysisException for following cases: 1. Generator is nested in expressions, e.g. SELECT explode(list) + 1 FROM tbl 2. more than one Generator is found in projectList, e.g. SELECT explode(list), explode(list) FROM tbl 3. Generator is found in other operators that are not Project or Generate, e.g. SELECT * FROM tbl SORT BY explode(list)

Extracts WindowExpressions from the projectList of a Project operator and aggregateExpressions of an Aggregate operator and creates individual Window operators for every distinct WindowSpecDefinition.

This rule handles three cases:

• A Project having WindowExpressions in its projectList;
• An Aggregate having WindowExpressions in its aggregateExpressions.
• A Filter->Aggregate pattern representing GROUP BY with a HAVING clause and the Aggregate has WindowExpressions in its aggregateExpressions. Note: If there is a GROUP BY clause in the query, aggregations and corresponding filters (expressions in the HAVING clause) should be evaluated before any WindowExpression. If a query has SELECT DISTINCT, the DISTINCT part should be evaluated after all WindowExpressions.

Note: ResolveLateralColumnAliasReference rule is applied before this rule. To guarantee this order, we make sure this rule applies only when the Project or Aggregate doesn't contain any LATERAL_COLUMN_ALIAS_REFERENCE.

For every case, the transformation works as follows: 1. For a list of Expressions (a projectList or an aggregateExpressions), partitions it two lists of Expressions, one for all WindowExpressions and another for all regular expressions. 2. For all WindowExpressions, groups them based on their WindowSpecDefinitions and WindowFunctionTypes. 3. For every distinct WindowSpecDefinition and WindowFunctionType, creates a Window operator and inserts it into the plan tree.

Correctly handle null primitive inputs for UDF by adding extra If expression to do the null check. When user defines a UDF with primitive parameters, there is no way to tell if the primitive parameter is null or not, so here we assume the primitive input is null-propagatable and we should return null if the input is null.

A rule to handle special commands that need to be notified when analysis is done. This rule should run after all other analysis rules are run.

Checks whether a function identifier referenced by an UnresolvedFunction is defined in the function registry. Note that this rule doesn't try to resolve the UnresolvedFunction. It only performs simple existence check according to the function identifier to quickly identify undefined functions without triggering relation resolution, which may incur potentially expensive partition/schema discovery process in some cases. In order to avoid duplicate external functions lookup, the external function identifier will store in the local hash set externalFunctionNameSet.

Extract non-session catalog and identifier from a multi-part identifier.

This rule finds aggregate expressions that are not in an aggregate operator. For example, those in a HAVING clause or ORDER BY clause. These expressions are pushed down to the underlying aggregate operator and then projected away after the original operator.

We need to make sure the expressions all fully resolved before looking for aggregate functions and group by expressions from them.

For Add: 1. if both side are interval, stays the same; 2. else if one side is date and the other is interval, turns it to DateAddInterval; 3. else if one side is interval, turns it to TimeAdd; 4. else if one side is date, turns it to DateAdd ; 5. else stays the same.

For Subtract: 1. if both side are interval, stays the same; 2. else if the left side is date and the right side is interval, turns it to -r); 3. else if the right side is an interval, turns it to -r); 4. else if one side is timestamp, turns it to SubtractTimestamps; 5. else if the right side is date, turns it to DateDiff/SubtractDates; 6. else if the left side is date, turns it to DateSub; 7. else turns it to stays the same.

For Multiply: 1. If one side is interval, turns it to MultiplyInterval; 2. otherwise, stays the same.

For Divide: 1. If the left side is interval, turns it to DivideInterval; 2. otherwise, stays the same.

Resolve the encoders for the UDF by explicitly given the attributes. We give the attributes explicitly in order to handle the case where the data type of the input value is not the same with the internal schema of the encoder, which could cause data loss. For example, the encoder should not cast the input value to Decimal(38, 18) if the actual data type is Decimal(30, 0).

The resolved encoders then will be used to deserialize the internal row to Scala value.

Replaces UnresolvedFunctionNames with concrete LogicalPlans. Replaces UnresolvedFunctions with concrete Expressions. Replaces UnresolvedGenerators with concrete Expressions. Replaces UnresolvedTableValuedFunctions with concrete LogicalPlans.

Rewrites table generating expressions that either need one or more of the following in order to be resolved:

• concrete attribute references for their output.
• to be relocated from a SELECT clause (i.e. from a Project) into a Generate).

Names for the output Attributes are extracted from Alias or MultiAlias expressions that wrap the Generator.

In many dialects of SQL it is valid to use ordinal positions in order/sort by and group by clauses. This rule is to convert ordinal positions to the corresponding expressions in the select list. This support is introduced in Spark 2.0.

- When the sort references or group by expressions are not integer but foldable expressions, just ignore them. - When spark.sql.orderByOrdinal/spark.sql.groupByOrdinal is set to false, ignore the position numbers too.

Before the release of Spark 2.0, the literals in order/sort by and group by clauses have no effect on the results.

Resolves columns of an output table from the data in a logical plan. This rule will:

- Reorder columns when the write is by name - Insert casts when data types do not match - Insert aliases when column names do not match - Detect plans that are not compatible with the output table and throw AnalysisException

This rule resolves SQL function expressions. It pulls out function inputs and place them in a separate Project node below the operator and replace the SQL function with its actual function body. SQL function expressions in Aggregate are handled in a special way. Non-aggregated SQL functions in the aggregate expressions of an Aggregate need to be pulled out into a Project above the Aggregate before replacing the SQL function expressions with actual function bodies. For example:

Before: Aggregate [c1] [foo(c1), foo(max(c2)), sum(foo(c2)) AS sum] +- Relation [c1, c2]

After: Project [foo(c1), foo(max_c2), sum] +- Aggregate [c1] [c1, max(c2) AS max_c2, sum(foo(c2)) AS sum] +- Relation [c1, c2]

This rule resolves and rewrites subqueries inside expressions.

Note: CTEs are handled in CTESubstitution.

Extract session catalog and identifier from a multi-part identifier.

A strategy that is run once and idempotent.