Packages

class NameScope extends SQLConfHelper

The NameScope is used to control the resolution of names (table, column, alias identifiers). It's a part of the Resolver's state, and is used to manage the output of SQL query/DataFrame program operators.

The NameScope output is immutable. If it's necessary to update the output, NameScopeStack methods are used (overwriteTop or withNewScope). The NameScope is always used through the NameScopeStack.

The resolution of identifiers is case-insensitive.

Name resolution priority is as follows:

  1. Resolution of local references:
    • column reference
    • struct field or map key reference 2. Resolution of lateral column aliases (if enabled).

For example, in a query like:

SELECT 1 AS col1, col1 FROM VALUES (2)

Because column resolution has a higher priority than LCA resolution, the result will be [1, 2] and not [1, 1].

Approximate tree of NameScope manipulations is shown in the following example:

CREATE TABLE IF NOT EXISTS t1 (col1 INT, col2 INT, col3 STRING);

SELECT
  col1, col2 as alias1
FROM
  (SELECT * FROM VALUES (1, 2))
  UNION
  (SELECT t2.col1, t2.col2 FROM (SELECT col1, col2 FROM t1) AS t2)
;

->

unionAttributes = withNewScope {
  lhsOutput = withNewScope {
    expandedStar = withNewScope {
      scope.overwriteTop(localRelation.output)
      scope.expandStar(star)
    }
    scope.overwriteTop(expandedStar)
    scope.output
  }
  rhsOutput = withNewScope {
    subqueryAttributes = withNewScope {
      scope.overwriteTop(t1.output)
      scope.overwriteTop(prependQualifier(scope.output, "t2"))
      [scope.matchMultiPartName("t2", "col1"), scope.matchMultiPartName("t2", "col2")]
    }
    scope.overwriteTop(subqueryAttributes)
    scope.output
  }
  scope.overwriteTop(coerce(lhsOutput, rhsOutput))
  [scope.matchMultiPartName("col1"), alias(scope.matchMultiPartName("col2"), "alias1")]
}
scope.overwriteTop(unionAttributes)
Linear Supertypes
SQLConfHelper, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. NameScope
  2. SQLConfHelper
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new NameScope(output: Seq[Attribute] = Seq.empty)

    output

    These are the attributes visible for lookups in the current scope. These may be:

    • Transformed outputs of lower scopes (e.g. type-coerced outputs of Union's children).
    • Output of a current operator that is being resolved (leaf nodes like Relations).

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  6. def conf: SQLConf

    The active config object within the current scope.

    The active config object within the current scope. See SQLConf.get for more information.

    Definition Classes
    SQLConfHelper
  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  9. def expandStar(unresolvedStar: UnresolvedStar): Seq[NamedExpression]

    Expand the UnresolvedStar.

    Expand the UnresolvedStar. The expected use case for this method is star expansion inside Project.

    Star without a target:

    -- Here the star will be expanded to [a, b, c].
    SELECT * FROM VALUES (1, 2, 3) AS t(a, b, c);

    Star with a multipart name target:

    USE CATALOG catalog1;
    USE DATABASE database1;
    
    CREATE TABLE IF NOT EXISTS table1 (col1 INT, col2 INT);
    
    -- Here the star will be expanded to [col1, col2].
    SELECT catalog1.database1.table1.* FROM catalog1.database1.table1;

    Star with a struct target:

    -- Here the star will be expanded to [field1, field2].
    SELECT d.* FROM VALUES (named_struct('field1', 1, 'field2', 2)) AS t(d);

    Star as an argument to a function:

    -- Here the star will be expanded to [col1, col2, col3] and those would be passed as
    -- arguments to `concat_ws`.
    SELECT concat_ws('', *) AS result FROM VALUES (1, 2, 3);

    Also, see UnresolvedStarBase.expandStar for more details.

  10. def findAttributesByName(name: String): Seq[Attribute]

    Find attributes in this NameScope that match a provided one-part name.

    Find attributes in this NameScope that match a provided one-part name.

    This method is simpler and more lightweight than resolveMultipartName, because here we just return all the attributes matched by the one-part name. This is only suitable for situations where name _resolution_ is not required (e.g. accessing struct fields from the lower operator's output).

    For example, this method is used to look up attributes to match a specific View schema. See ExpressionResolver.resolveGetViewColumnByNameAndOrdinal for more info on view column lookup.

    We are relying on a simple IdentifierMap to perform that work, since we just need to match one-part name from the lower operator's output here.

  11. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  12. def hasAttributeWithId(expressionId: ExprId): Boolean

    Check if output contains attributes with expressionId.

    Check if output contains attributes with expressionId. This is used to disable missing attribute propagation for DataFrames, because we don't support it yet.

  13. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  14. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  15. lazy val lcaRegistry: LateralColumnAliasRegistry
  16. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  17. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  18. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  19. val output: Seq[Attribute]
  20. def resolveMultipartName(multipartName: Seq[String], canLaterallyReferenceColumn: Boolean = true): NameTarget

    Resolve multipart name into a NameTarget.

    Resolve multipart name into a NameTarget. NameTarget's candidates may contain simple AttributeReferences if it's a column or alias, or ExtractValue expressions if it's a struct field, map value or array value. The aliasName will optionally be set to the proposed alias name for the value extracted from a struct, map or array.

    Example that demonstrates those major use-cases:

    CREATE TABLE IF NOT EXISTS t (
      col1 INT,
      col2 STRUCT<field: INT>,
      col3 STRUCT<struct: STRUCT<field: INT>>,
      col4 MAP<STRING: INT>,
      col5 STRING
    );
    
    -- For the SELECT below the top Project list will be resolved using this method like this:
    -- AttributeReference(col1),
    -- AttributeReference(a),
    -- GetStructField(col2, field),
    -- GetStructField(GetStructField(col3, struct), field),
    -- GetMapValue(col4, key)
    SELECT
      col1, a, col2.field, col3.struct.field, col4.key
    FROM
      (SELECT *, col5 AS a FROM t);

    Since there can be several expressions that matched the same multipart name, this method may return a NameTarget with the following candidates: - 0 values: No matched expressions - 1 value: Unique expression matched - 1+ values: Ambiguity, several expressions matched

    Some examples of ambiguity:

    CREATE TABLE IF NOT EXISTS t1 (c1 INT, c2 INT);
    CREATE TABLE IF NOT EXISTS t2 (c2 INT, c3 INT);
    
    -- Identically named columns from different tables.
    -- This will fail with AMBIGUOUS_REFERENCE error.
    SELECT c2 FROM t1, t2;
    CREATE TABLE IF NOT EXISTS foo (c1 INT);
    CREATE TABLE IF NOT EXISTS bar (foo STRUCT<c1: INT>);
    
    -- Ambiguity between a column in a table and a field in a struct.
    -- This will succeed, and column will win over the struct field.
    SELECT foo.c1 FROM foo, bar;

    The candidates are deduplicated by expression ID (not by attribute name!):

    CREATE TABLE IF NOT EXISTS t1 (col1 STRING);
    
    -- No ambiguity here, since we are selecting the same column (same expression ID).
    SELECT col1 FROM (SELECT col1, col1 FROM t);

    The case of the multipartName takes precedence over the original name case, so the candidates will have names that are case-identical to the multipartName:

    CREATE TABLE IF NOT EXISTS t1 (col1 STRING);
    
    -- The output schema of this query is [COL1], despite the fact that the column is in
    -- lower-case.
    SELECT COL1 FROM t;

    We are relying on the AttributeSeq to perform that work, since it requires complex resolution logic involving nested field extraction and multipart name matching.

    Also, see AttributeSeq.resolve for more details.

  21. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  22. def toString(): String
    Definition Classes
    AnyRef → Any
  23. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  24. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  25. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  26. def withSQLConf[T](pairs: (String, String)*)(f: => T): T

    Sets all SQL configurations specified in pairs, calls f, and then restores all SQL configurations.

    Sets all SQL configurations specified in pairs, calls f, and then restores all SQL configurations.

    Attributes
    protected
    Definition Classes
    SQLConfHelper

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from SQLConfHelper

Inherited from AnyRef

Inherited from Any

Ungrouped