Packages

o

org.apache.spark.sql.catalyst.util

ResolveDefaultColumns

object ResolveDefaultColumns extends QueryErrorsBase with ResolveDefaultColumnsUtils with SQLConfHelper with Logging

This object contains fields to help process DEFAULT columns.

Linear Supertypes
Logging, SQLConfHelper, ResolveDefaultColumnsUtils, QueryErrorsBase, DataTypeErrorsBase, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ResolveDefaultColumns
  2. Logging
  3. SQLConfHelper
  4. ResolveDefaultColumnsUtils
  5. QueryErrorsBase
  6. DataTypeErrorsBase
  7. AnyRef
  8. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Type Members

  1. implicit class LogStringContext extends AnyRef
    Definition Classes
    Logging

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. val CURRENT_DEFAULT_COLUMN_METADATA_KEY: String
    Definition Classes
    ResolveDefaultColumnsUtils
  5. val CURRENT_DEFAULT_COLUMN_NAME: String
  6. val EXISTS_DEFAULT_COLUMN_METADATA_KEY: String
    Definition Classes
    ResolveDefaultColumnsUtils
  7. def analyze(colName: String, dataType: DataType, defaultValue: DefaultValue, statementType: String): Expression

    Analyzes the connector default value.

    Analyzes the connector default value.

    If the default value is defined as a connector expression, Spark first attempts to convert it to a Catalyst expression. If conversion fails but a SQL string is provided, the SQL is parsed instead. If only a SQL string is present, it is parsed directly.

    returns

    the result of the analysis and constant-folding operation

  8. def analyze(colName: String, dataType: DataType, defaultSQL: String, statementType: String): Expression

    Parses and analyzes the DEFAULT column SQL string, returning an error upon failure.

    Parses and analyzes the DEFAULT column SQL string, returning an error upon failure.

    returns

    Result of the analysis and constant-folding operation.

  9. def analyze(field: StructField, statementType: String, metadataKey: String = CURRENT_DEFAULT_COLUMN_METADATA_KEY): Expression

    Parses and analyzes the DEFAULT column text in field, returning an error upon failure.

    Parses and analyzes the DEFAULT column text in field, returning an error upon failure.

    field

    represents the DEFAULT column value whose "default" metadata to parse and analyze.

    statementType

    which type of statement we are running, such as INSERT; useful for errors.

    metadataKey

    which key to look up from the column metadata; generally either CURRENT_DEFAULT_COLUMN_METADATA_KEY or EXISTS_DEFAULT_COLUMN_METADATA_KEY.

    returns

    Result of the analysis and constant-folding operation.

  10. def applyExistenceDefaultValuesToRow(schema: StructType, row: InternalRow, bitmask: Array[Boolean]): Unit

    Updates a subset of columns in the row with default values from the metadata in the schema.

  11. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  12. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  13. def coerceDefaultValue(analyzed: Expression, dataType: DataType, statementType: String, colName: String, defaultSQL: String): Expression

    Returns the result of type coercion from analyzed to dataType, or throws an error if the expression is not coercible.

  14. def conf: SQLConf

    The active config object within the current scope.

    The active config object within the current scope. See SQLConf.get for more information.

    Definition Classes
    SQLConfHelper
  15. def constantFoldCurrentDefaultsToExistDefaults(tableSchema: StructType, statementType: String): StructType

    Finds "current default" expressions in CREATE/REPLACE TABLE columns and constant-folds them.

    Finds "current default" expressions in CREATE/REPLACE TABLE columns and constant-folds them.

    The results are stored in the "exists default" metadata of the same columns. For example, in the event of this statement:

    CREATE TABLE T(a INT, b INT DEFAULT 5 + 5)

    This method constant-folds the "current default" value, stored in the CURRENT_DEFAULT metadata of the "b" column, to "10", storing the result in the "exists default" value within the EXISTS_DEFAULT metadata of that same column. Meanwhile the "current default" metadata of this "b" column retains its original value of "5 + 5".

    The reason for constant-folding the EXISTS_DEFAULT is to make the end-user visible behavior the same, after executing an ALTER TABLE ADD COLUMNS command with DEFAULT value, as if the system had performed an exhaustive backfill of the provided value to all previously existing rows in the table instead. We choose to avoid doing such a backfill because it would be a time-consuming and costly operation. Instead, we elect to store the EXISTS_DEFAULT in the column metadata for future reference when querying data out of the data source. In turn, each data source then takes responsibility to provide the constant-folded value in the EXISTS_DEFAULT metadata for such columns where the value is not present in storage.

    tableSchema

    represents the names and types of the columns of the statement to process.

    statementType

    name of the statement being processed, such as INSERT; useful for errors.

    returns

    a copy of tableSchema with field metadata updated with the constant-folded values.

  16. def containsExplicitDefaultColumn(expr: Expression): Boolean

    Returns true if the given expression contains an explicit DEFAULT column reference.

  17. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  18. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  19. def existenceDefaultValues(schema: StructType): Array[Any]

    These define existence default values for the struct fields for efficiency purposes.

    These define existence default values for the struct fields for efficiency purposes. The caller should avoid using such methods in a loop for efficiency.

  20. def existenceDefaultsBitmask(schema: StructType): Array[Boolean]
  21. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  22. def getDefaultValueExprOrNullLit(attr: Attribute, useNullAsDefault: Boolean): Option[NamedExpression]

    Generates the expression of the default value for the given attribute.

    Generates the expression of the default value for the given attribute. If there is no user-specified default value for this attribute, returns null literal if useNullAsDefault is true and the attribute is nullable.

  23. def getDefaultValueExprOrNullLit(field: StructField, useNullAsDefault: Boolean): Option[NamedExpression]

    Generates the expression of the default value for the given field.

    Generates the expression of the default value for the given field. If there is no user-specified default value for this field, returns null literal if useNullAsDefault is true and the field is nullable.

  24. def getDefaultValueExprOrNullLit(attr: Attribute): Expression

    Generates the expression of the default value for the given attribute.

    Generates the expression of the default value for the given attribute. If there is no user-specified default value for this attribute and the attribute is nullable, returns null literal, otherwise an exception is thrown.

  25. def getDefaultValueExprOrNullLit(field: StructField): Expression

    Generates the expression of the default value for the given field.

    Generates the expression of the default value for the given field. If there is no user-specified default value for this field and the field is nullable, returns null literal, otherwise an exception is thrown.

  26. def getDescribeMetadata(schema: StructType): Seq[(String, String, String)]

    If any fields in a schema have default values, appends them to the result.

  27. def getExistenceDefaultValues(schema: StructType): Array[Any]

    Parses the text representing constant-folded default column literal values.

    Parses the text representing constant-folded default column literal values. These are known as "existence" default values because each one is the constant-folded result of the original default value first assigned to the column at table/column creation time. When scanning a field from any data source, if the corresponding value is not present in storage, the output row returns this "existence" default value instead of NULL.

    returns

    a sequence of either (1) NULL, if the column had no default value, or (2) an object of Any type suitable for assigning into a row using the InternalRow.update method.

  28. def getExistenceDefaultsBitmask(schema: StructType): Array[Boolean]

    Returns an array of boolean values equal in size to the result of getExistenceDefaultValues above, for convenience.

  29. def getQueryContext(context: QueryContext): Array[QueryContext]
    Definition Classes
    DataTypeErrorsBase
  30. def getSummary(sqlContext: QueryContext): String
    Definition Classes
    DataTypeErrorsBase
  31. def hasExistenceDefaultValues(schema: StructType): Boolean
  32. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  33. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  34. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  35. def isExplicitDefaultColumn(col: UnresolvedAttribute): Boolean

    Returns true if the unresolved column is an explicit DEFAULT column reference.

  36. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  37. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  38. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  39. def logDebug(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  40. def logDebug(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  41. def logDebug(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  42. def logDebug(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  43. def logError(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  44. def logError(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  45. def logError(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  46. def logError(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  47. def logInfo(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  48. def logInfo(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  49. def logInfo(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  50. def logInfo(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  51. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  52. def logTrace(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  53. def logTrace(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  54. def logTrace(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  55. def logTrace(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  56. def logWarning(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  57. def logWarning(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  58. def logWarning(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  59. def logWarning(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  60. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  61. def normalizeFieldName(str: String): String

    Normalizes a schema field name suitable for use in looking up into maps keyed by schema field names.

    Normalizes a schema field name suitable for use in looking up into maps keyed by schema field names.

    str

    the field name to normalize

    returns

    the normalized result

  62. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  63. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  64. def ordinalNumber(i: Int): String
    Definition Classes
    QueryErrorsBase
  65. def quoteByDefault(elem: String): String
    Attributes
    protected
    Definition Classes
    DataTypeErrorsBase
  66. def resetExistenceDefaultsBitmask(schema: StructType, bitmask: Array[Boolean]): Unit

    Resets the elements of the array initially returned from getExistenceDefaultsBitmask above.

    Resets the elements of the array initially returned from getExistenceDefaultsBitmask above. Afterwards, set element(s) to false before calling applyExistenceDefaultValuesToRow below.

  67. def resolveColumnDefaultInAssignmentValue(key: Expression, value: Expression, invalidColumnDefaultException: => Throwable): Expression

    Resolves the column "DEFAULT" in UPDATE/MERGE assignment value expression if the following conditions are met: 1.

    Resolves the column "DEFAULT" in UPDATE/MERGE assignment value expression if the following conditions are met: 1. The assignment value expression is a single UnresolvedAttribute with name "DEFAULT". This means key = DEFAULT is allowed but key = DEFAULT + 1 is not. 2. The assignment key expression is a top-level column. This means col = DEFAULT is allowed but col.field = DEFAULT is not.

    The column "DEFAULT" will be resolved to the default value expression defined for the column of the assignment key.

  68. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  69. def toDSOption(option: String): String
    Definition Classes
    DataTypeErrorsBase
  70. def toSQLConf(conf: String): String
    Definition Classes
    DataTypeErrorsBase
  71. def toSQLConfVal(conf: String): String
    Definition Classes
    QueryErrorsBase
  72. def toSQLExpr(e: Expression): String
    Definition Classes
    QueryErrorsBase
  73. def toSQLId(parts: Seq[String]): String
    Definition Classes
    DataTypeErrorsBase
  74. def toSQLId(parts: String): String
    Definition Classes
    DataTypeErrorsBase
  75. def toSQLStmt(text: String): String
    Definition Classes
    DataTypeErrorsBase
  76. def toSQLType(t: AbstractDataType): String
    Definition Classes
    DataTypeErrorsBase
  77. def toSQLType(text: String): String
    Definition Classes
    DataTypeErrorsBase
  78. def toSQLValue(v: Any, t: DataType): String
    Definition Classes
    QueryErrorsBase
  79. def toSQLValue(value: Double): String
    Definition Classes
    DataTypeErrorsBase
  80. def toSQLValue(value: Float): String
    Definition Classes
    DataTypeErrorsBase
  81. def toSQLValue(value: Long): String
    Definition Classes
    DataTypeErrorsBase
  82. def toSQLValue(value: Int): String
    Definition Classes
    DataTypeErrorsBase
  83. def toSQLValue(value: Short): String
    Definition Classes
    DataTypeErrorsBase
  84. def toSQLValue(value: UTF8String): String
    Definition Classes
    DataTypeErrorsBase
  85. def toSQLValue(value: String): String
    Definition Classes
    DataTypeErrorsBase
  86. def toString(): String
    Definition Classes
    AnyRef → Any
  87. def validateCatalogForDefaultValue(columns: Seq[ColumnDefinition], catalog: TableCatalog, ident: Identifier): Unit
  88. def validateDefaultValueExpr(default: DefaultValueExpression, statement: String, colName: String, targetType: DataType): Unit
  89. def validateTableProviderForDefaultValue(schema: StructType, tableProvider: Option[String], statementType: String, addNewColumnToExistingTable: Boolean): Unit
  90. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  91. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  92. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  93. def withLogContext(context: Map[String, String])(body: => Unit): Unit
    Attributes
    protected
    Definition Classes
    Logging
  94. def withSQLConf[T](pairs: (String, String)*)(f: => T): T

    Sets all SQL configurations specified in pairs, calls f, and then restores all SQL configurations.

    Sets all SQL configurations specified in pairs, calls f, and then restores all SQL configurations.

    Attributes
    protected
    Definition Classes
    SQLConfHelper
  95. object BuiltInFunctionCatalog extends FunctionCatalog

    This is a FunctionCatalog for performing analysis using built-in functions only.

    This is a FunctionCatalog for performing analysis using built-in functions only. It is a helper for the DefaultColumnAnalyzer above.

  96. object DefaultColumnAnalyzer extends Analyzer

    This is an Analyzer for processing default column values using built-in functions only.

  97. object DefaultColumnOptimizer extends Optimizer

    This is an Optimizer for convert default column expressions to foldable literals.

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from Logging

Inherited from SQLConfHelper

Inherited from ResolveDefaultColumnsUtils

Inherited from QueryErrorsBase

Inherited from DataTypeErrorsBase

Inherited from AnyRef

Inherited from Any

Ungrouped