class NameScope extends SQLConfHelper
The NameScope is used to control the resolution of names (table, column, alias identifiers). It's a part of the Resolver's state, and is used to manage the output of SQL query/DataFrame program operators.
The NameScope output is immutable. If it's necessary to update the output, NameScopeStack methods are used (overwriteTop or withNewScope). The NameScope is always used through the NameScopeStack.
The resolution of identifiers is case-insensitive.
Name resolution priority is as follows:
- Resolution of local references:
- column reference
- struct field or map key reference 2. Resolution of lateral column aliases (if enabled).
For example, in a query like:
SELECT 1 AS col1, col1 FROM VALUES (2)
Because column resolution has a higher priority than LCA resolution, the result will be [1, 2] and not [1, 1].
Approximate tree of NameScope manipulations is shown in the following example:
CREATE TABLE IF NOT EXISTS t1 (col1 INT, col2 INT, col3 STRING); SELECT col1, col2 as alias1 FROM (SELECT * FROM VALUES (1, 2)) UNION (SELECT t2.col1, t2.col2 FROM (SELECT col1, col2 FROM t1) AS t2) ;
->
unionAttributes = withNewScope {
lhsOutput = withNewScope {
expandedStar = withNewScope {
scope.overwriteTop(localRelation.output)
scope.expandStar(star)
}
scope.overwriteTop(expandedStar)
scope.output
}
rhsOutput = withNewScope {
subqueryAttributes = withNewScope {
scope.overwriteTop(t1.output)
scope.overwriteTop(prependQualifier(scope.output, "t2"))
[scope.matchMultiPartName("t2", "col1"), scope.matchMultiPartName("t2", "col2")]
}
scope.overwriteTop(subqueryAttributes)
scope.output
}
scope.overwriteTop(coerce(lhsOutput, rhsOutput))
[scope.matchMultiPartName("col1"), alias(scope.matchMultiPartName("col2"), "alias1")]
}
scope.overwriteTop(unionAttributes)- Alphabetic
- By Inheritance
- NameScope
- SQLConfHelper
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new NameScope(output: Seq[Attribute] = Seq.empty)
- output
These are the attributes visible for lookups in the current scope. These may be:
- Transformed outputs of lower scopes (e.g. type-coerced outputs of Union's children).
- Output of a current operator that is being resolved (leaf nodes like Relations).
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- def conf: SQLConf
The active config object within the current scope.
The active config object within the current scope. See SQLConf.get for more information.
- Definition Classes
- SQLConfHelper
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def expandStar(unresolvedStar: UnresolvedStar): Seq[NamedExpression]
Expand the UnresolvedStar.
Expand the UnresolvedStar. The expected use case for this method is star expansion inside Project.
Star without a target:
-- Here the star will be expanded to [a, b, c]. SELECT * FROM VALUES (1, 2, 3) AS t(a, b, c);
Star with a multipart name target:
USE CATALOG catalog1; USE DATABASE database1; CREATE TABLE IF NOT EXISTS table1 (col1 INT, col2 INT); -- Here the star will be expanded to [col1, col2]. SELECT catalog1.database1.table1.* FROM catalog1.database1.table1;
Star with a struct target:
-- Here the star will be expanded to [field1, field2]. SELECT d.* FROM VALUES (named_struct('field1', 1, 'field2', 2)) AS t(d);
Star as an argument to a function:
-- Here the star will be expanded to [col1, col2, col3] and those would be passed as -- arguments to `concat_ws`. SELECT concat_ws('', *) AS result FROM VALUES (1, 2, 3);
Also, see UnresolvedStarBase.expandStar for more details.
- def findAttributesByName(name: String): Seq[Attribute]
Find attributes in this NameScope that match a provided one-part
name.Find attributes in this NameScope that match a provided one-part
name.This method is simpler and more lightweight than resolveMultipartName, because here we just return all the attributes matched by the one-part
name. This is only suitable for situations where name _resolution_ is not required (e.g. accessing struct fields from the lower operator's output).For example, this method is used to look up attributes to match a specific View schema. See ExpressionResolver.resolveGetViewColumnByNameAndOrdinal for more info on view column lookup.
We are relying on a simple IdentifierMap to perform that work, since we just need to match one-part name from the lower operator's output here.
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hasAttributeWithId(expressionId: ExprId): Boolean
Check if
outputcontains attributes withexpressionId.Check if
outputcontains attributes withexpressionId. This is used to disable missing attribute propagation for DataFrames, because we don't support it yet. - def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- lazy val lcaRegistry: LateralColumnAliasRegistry
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- val output: Seq[Attribute]
- def resolveMultipartName(multipartName: Seq[String], canLaterallyReferenceColumn: Boolean = true): NameTarget
Resolve multipart name into a NameTarget.
Resolve multipart name into a NameTarget. NameTarget's
candidatesmay contain simple AttributeReferences if it's a column or alias, or ExtractValue expressions if it's a struct field, map value or array value. ThealiasNamewill optionally be set to the proposed alias name for the value extracted from a struct, map or array.Example that demonstrates those major use-cases:
CREATE TABLE IF NOT EXISTS t ( col1 INT, col2 STRUCT<field: INT>, col3 STRUCT<struct: STRUCT<field: INT>>, col4 MAP<STRING: INT>, col5 STRING ); -- For the SELECT below the top Project list will be resolved using this method like this: -- AttributeReference(col1), -- AttributeReference(a), -- GetStructField(col2, field), -- GetStructField(GetStructField(col3, struct), field), -- GetMapValue(col4, key) SELECT col1, a, col2.field, col3.struct.field, col4.key FROM (SELECT *, col5 AS a FROM t);
Since there can be several expressions that matched the same multipart name, this method may return a NameTarget with the following
candidates: - 0 values: No matched expressions - 1 value: Unique expression matched - 1+ values: Ambiguity, several expressions matchedSome examples of ambiguity:
CREATE TABLE IF NOT EXISTS t1 (c1 INT, c2 INT); CREATE TABLE IF NOT EXISTS t2 (c2 INT, c3 INT); -- Identically named columns from different tables. -- This will fail with AMBIGUOUS_REFERENCE error. SELECT c2 FROM t1, t2;CREATE TABLE IF NOT EXISTS foo (c1 INT); CREATE TABLE IF NOT EXISTS bar (foo STRUCT<c1: INT>); -- Ambiguity between a column in a table and a field in a struct. -- This will succeed, and column will win over the struct field. SELECT foo.c1 FROM foo, bar;
The candidates are deduplicated by expression ID (not by attribute name!):
CREATE TABLE IF NOT EXISTS t1 (col1 STRING); -- No ambiguity here, since we are selecting the same column (same expression ID). SELECT col1 FROM (SELECT col1, col1 FROM t);
The case of the
multipartNametakes precedence over the original name case, so the candidates will have names that are case-identical to themultipartName:CREATE TABLE IF NOT EXISTS t1 (col1 STRING); -- The output schema of this query is [COL1], despite the fact that the column is in -- lower-case. SELECT COL1 FROM t;
We are relying on the AttributeSeq to perform that work, since it requires complex resolution logic involving nested field extraction and multipart name matching.
Also, see AttributeSeq.resolve for more details.
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- def withSQLConf[T](pairs: (String, String)*)(f: => T): T
Sets all SQL configurations specified in
pairs, callsf, and then restores all SQL configurations.Sets all SQL configurations specified in
pairs, callsf, and then restores all SQL configurations.- Attributes
- protected
- Definition Classes
- SQLConfHelper
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)