Packages

abstract class MergeIntoCommandBase extends LogicalPlan with LeafRunnableCommand with DeltaCommand with DeltaLogging with PredicateHelper with ImplicitMetadataOperation with MergeIntoMaterializeSource

Linear Supertypes
MergeIntoMaterializeSource, DeltaSparkPlanUtils, ImplicitMetadataOperation, PredicateHelper, AliasHelper, DeltaCommand, DeltaLogging, DatabricksLogging, DeltaProgressReporter, LeafRunnableCommand, LeafLike[LogicalPlan], RunnableCommand, Command, LogicalPlan, Logging, QueryPlanConstraints, ConstraintHelper, LogicalPlanDistinctKeys, LogicalPlanStats, AnalysisHelper, QueryPlan[LogicalPlan], SQLConfHelper, TreeNode[LogicalPlan], WithOrigin, TreePatternBits, Product, Equals, AnyRef, Any
Known Subclasses
MergeIntoCommand
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. MergeIntoCommandBase
  2. MergeIntoMaterializeSource
  3. DeltaSparkPlanUtils
  4. ImplicitMetadataOperation
  5. PredicateHelper
  6. AliasHelper
  7. DeltaCommand
  8. DeltaLogging
  9. DatabricksLogging
  10. DeltaProgressReporter
  11. LeafRunnableCommand
  12. LeafLike
  13. RunnableCommand
  14. Command
  15. LogicalPlan
  16. Logging
  17. QueryPlanConstraints
  18. ConstraintHelper
  19. LogicalPlanDistinctKeys
  20. LogicalPlanStats
  21. AnalysisHelper
  22. QueryPlan
  23. SQLConfHelper
  24. TreeNode
  25. WithOrigin
  26. TreePatternBits
  27. Product
  28. Equals
  29. AnyRef
  30. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new MergeIntoCommandBase()

Type Members

  1. type PlanOrExpression = Either[LogicalPlan, Expression]
    Definition Classes
    DeltaSparkPlanUtils

Abstract Value Members

  1. abstract def canEqual(that: Any): Boolean
    Definition Classes
    Equals
  2. abstract val condition: Expression
  3. abstract val matchedClauses: Seq[DeltaMergeIntoMatchedClause]
  4. abstract val migratedSchema: Option[StructType]
  5. abstract val notMatchedBySourceClauses: Seq[DeltaMergeIntoNotMatchedBySourceClause]
  6. abstract val notMatchedClauses: Seq[DeltaMergeIntoNotMatchedClause]
  7. abstract def productArity: Int
    Definition Classes
    Product
  8. abstract def productElement(n: Int): Any
    Definition Classes
    Product
  9. abstract def runMerge(spark: SparkSession): Seq[Row]
    Attributes
    protected
  10. abstract val source: LogicalPlan
  11. abstract val target: LogicalPlan
  12. abstract val targetFileIndex: TahoeFileIndex

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. lazy val allAttributes: AttributeSeq
    Definition Classes
    QueryPlan
  5. def analyzed: Boolean
    Definition Classes
    AnalysisHelper
  6. def apply(number: Int): TreeNode[_]
    Definition Classes
    TreeNode
  7. def argString(maxFields: Int): String
    Definition Classes
    TreeNode
  8. def asCode: String
    Definition Classes
    TreeNode
  9. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  10. def assertNotAnalysisRule(): Unit
    Attributes
    protected
    Definition Classes
    AnalysisHelper
  11. val attempt: Int

    Track which attempt or retry it is in runWithMaterializedSourceAndRetries

    Track which attempt or retry it is in runWithMaterializedSourceAndRetries

    Attributes
    protected
    Definition Classes
    MergeIntoMaterializeSource
  12. lazy val baseMetrics: Map[String, SQLMetric]
  13. def buildBalancedPredicate(expressions: Seq[Expression], op: (Expression, Expression) => Expression): Expression
    Attributes
    protected
    Definition Classes
    PredicateHelper
  14. def buildBaseRelation(spark: SparkSession, txn: OptimisticTransaction, actionType: String, rootPath: Path, inputLeafFiles: Seq[String], nameToAddFileMap: Map[String, AddFile]): HadoopFsRelation

    Build a base relation of files that need to be rewritten as part of an update/delete/merge operation.

    Build a base relation of files that need to be rewritten as part of an update/delete/merge operation.

    Attributes
    protected
    Definition Classes
    DeltaCommand
  15. def buildTargetPlanWithFiles(spark: SparkSession, deltaTxn: OptimisticTransaction, files: Seq[AddFile], columnsToDrop: Seq[String]): LogicalPlan

    Builds a new logical plan to read the given files instead of the whole target table.

    Builds a new logical plan to read the given files instead of the whole target table. The plan returned has the same output columns (exprIds) as the target logical plan, so that existing update/insert expressions can be applied on this new plan. Unneeded non-partition columns may be dropped.

    Attributes
    protected
  16. def buildTargetPlanWithIndex(spark: SparkSession, deltaTxn: OptimisticTransaction, fileIndex: TahoeFileIndex, columnsToDrop: Seq[String]): LogicalPlan

    Builds a new logical plan to read the target table using the given fileIndex.

    Builds a new logical plan to read the target table using the given fileIndex. The plan returned has the same output columns (exprIds) as the target logical plan, so that existing update/insert expressions can be applied on this new plan.

    columnsToDrop

    unneeded non-partition columns to be dropped

    Attributes
    protected
  17. def canEvaluate(expr: Expression, plan: LogicalPlan): Boolean
    Attributes
    protected
    Definition Classes
    PredicateHelper
  18. def canEvaluateWithinJoin(expr: Expression): Boolean
    Attributes
    protected
    Definition Classes
    PredicateHelper
  19. val canMergeSchema: Boolean
    Definition Classes
    MergeIntoCommandBase → ImplicitMetadataOperation
  20. val canOverwriteSchema: Boolean
    Definition Classes
    MergeIntoCommandBase → ImplicitMetadataOperation
  21. final lazy val canonicalized: LogicalPlan
    Definition Classes
    QueryPlan
    Annotations
    @transient()
  22. def checkNonDeterministicSource(spark: SparkSession): Unit

    Throws an exception if merge metrics indicate that the source table changed between the first and the second source table scans.

    Throws an exception if merge metrics indicate that the source table changed between the first and the second source table scans.

    Attributes
    protected
  23. final def children: Seq[LogicalPlan]
    Definition Classes
    LeafLike
  24. def childrenResolved: Boolean
    Definition Classes
    LogicalPlan
  25. def clone(): LogicalPlan
    Definition Classes
    AnalysisHelper → TreeNode → AnyRef
  26. def collect[B](pf: PartialFunction[LogicalPlan, B]): Seq[B]
    Definition Classes
    TreeNode
  27. def collectFirst[In, Out](input: Iterable[In], recurse: (In) => Option[Out]): Option[Out]
    Attributes
    protected
    Definition Classes
    DeltaSparkPlanUtils
  28. def collectFirst[B](pf: PartialFunction[LogicalPlan, B]): Option[B]
    Definition Classes
    TreeNode
  29. def collectLeaves(): Seq[LogicalPlan]
    Definition Classes
    TreeNode
  30. def collectMergeStats(deltaTxn: OptimisticTransaction, materializeSourceReason: MergeIntoMaterializeSourceReason): MergeStats

    Collects the merge operation stats and metrics into a MergeStats object that can be recorded with recordDeltaEvent.

    Collects the merge operation stats and metrics into a MergeStats object that can be recorded with recordDeltaEvent. Merge stats should be collected after committing all new actions as metrics may still be updated during commit.

    Attributes
    protected
  31. def collectWithSubqueries[B](f: PartialFunction[LogicalPlan, B]): Seq[B]
    Definition Classes
    QueryPlan
  32. def conf: SQLConf
    Definition Classes
    SQLConfHelper
  33. lazy val constraints: ExpressionSet
    Definition Classes
    QueryPlanConstraints
  34. def constructIsNotNullConstraints(constraints: ExpressionSet, output: Seq[Attribute]): ExpressionSet
    Definition Classes
    ConstraintHelper
  35. final def containsAllPatterns(patterns: TreePattern*): Boolean
    Definition Classes
    TreePatternBits
  36. final def containsAnyPattern(patterns: TreePattern*): Boolean
    Definition Classes
    TreePatternBits
  37. lazy val containsChild: Set[TreeNode[_]]
    Definition Classes
    TreeNode
  38. final def containsPattern(t: TreePattern): Boolean
    Definition Classes
    TreePatternBits
    Annotations
    @inline()
  39. def copyTagsFrom(other: LogicalPlan): Unit
    Definition Classes
    TreeNode
  40. def createSetTransaction(sparkSession: SparkSession, deltaLog: DeltaLog, options: Option[DeltaOptions] = None): Option[SetTransaction]

    Returns SetTransaction if a valid app ID and version are present.

    Returns SetTransaction if a valid app ID and version are present. Otherwise returns an empty list.

    Attributes
    protected
    Definition Classes
    DeltaCommand
  41. lazy val deterministic: Boolean
    Definition Classes
    QueryPlan
  42. lazy val distinctKeys: Set[ExpressionSet]
    Definition Classes
    LogicalPlanDistinctKeys
  43. def doCanonicalize(): LogicalPlan
    Attributes
    protected
    Definition Classes
    QueryPlan
  44. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  45. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  46. def exists(f: (LogicalPlan) => Boolean): Boolean
    Definition Classes
    TreeNode
  47. final def expressions: Seq[Expression]
    Definition Classes
    QueryPlan
  48. def extractPredicatesWithinOutputSet(condition: Expression, outputSet: AttributeSet): Option[Expression]
    Attributes
    protected
    Definition Classes
    PredicateHelper
  49. def fastEquals(other: TreeNode[_]): Boolean
    Definition Classes
    TreeNode
  50. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  51. def find(f: (LogicalPlan) => Boolean): Option[LogicalPlan]
    Definition Classes
    TreeNode
  52. def findExpressionAndTrackLineageDown(exp: Expression, plan: LogicalPlan): Option[(Expression, LogicalPlan)]
    Definition Classes
    PredicateHelper
  53. def findFirstNonDeltaScan(source: LogicalPlan): Option[LogicalPlan]
    Attributes
    protected
    Definition Classes
    DeltaSparkPlanUtils
  54. def findFirstNonDeterministicChildNode(children: Seq[Expression]): Option[PlanOrExpression]
    Attributes
    protected
    Definition Classes
    DeltaSparkPlanUtils
  55. def findFirstNonDeterministicNode(child: Expression): Option[PlanOrExpression]
    Attributes
    protected
    Definition Classes
    DeltaSparkPlanUtils
  56. def findFirstNonDeterministicNode(plan: LogicalPlan): Option[PlanOrExpression]

    Returns a part of the plan that does not have a safe level of determinism.

    Returns a part of the plan that does not have a safe level of determinism. This is a conservative approximation of plan being a truly deterministic query.

    Attributes
    protected
    Definition Classes
    DeltaSparkPlanUtils
  57. def flatMap[A](f: (LogicalPlan) => TraversableOnce[A]): Seq[A]
    Definition Classes
    TreeNode
  58. def foreach(f: (LogicalPlan) => Unit): Unit
    Definition Classes
    TreeNode
  59. def foreachUp(f: (LogicalPlan) => Unit): Unit
    Definition Classes
    TreeNode
  60. def formattedNodeName: String
    Attributes
    protected
    Definition Classes
    QueryPlan
  61. def generateCandidateFileMap(basePath: Path, candidateFiles: Seq[AddFile]): Map[String, AddFile]

    Generates a map of file names to add file entries for operations where we will need to rewrite files such as delete, merge, update.

    Generates a map of file names to add file entries for operations where we will need to rewrite files such as delete, merge, update. We expect file names to be unique, because each file contains a UUID.

    Definition Classes
    DeltaCommand
  62. def generateTreeString(depth: Int, lastChildren: ArrayList[Boolean], append: (String) => Unit, verbose: Boolean, prefix: String, addSuffix: Boolean, maxFields: Int, printNodeId: Boolean, indent: Int): Unit
    Definition Classes
    TreeNode
  63. def getAliasMap(exprs: Seq[NamedExpression]): AttributeMap[Alias]
    Attributes
    protected
    Definition Classes
    AliasHelper
  64. def getAliasMap(plan: Aggregate): AttributeMap[Alias]
    Attributes
    protected
    Definition Classes
    AliasHelper
  65. def getAliasMap(plan: Project): AttributeMap[Alias]
    Attributes
    protected
    Definition Classes
    AliasHelper
  66. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  67. def getCommonTags(deltaLog: DeltaLog, tahoeId: String): Map[TagDefinition, String]
    Definition Classes
    DeltaLogging
  68. def getDefaultTreePatternBits: BitSet
    Attributes
    protected
    Definition Classes
    TreeNode
  69. def getDeltaLog(spark: SparkSession, path: Option[String], tableIdentifier: Option[TableIdentifier], operationName: String, hadoopConf: Map[String, String] = Map.empty): DeltaLog

    Utility method to return the DeltaLog of an existing Delta table referred by either the given path or

    Utility method to return the DeltaLog of an existing Delta table referred by either the given path or

    spark

    SparkSession reference to use.

    path

    Table location. Expects a non-empty tableIdentifier or path.

    tableIdentifier

    Table identifier. Expects a non-empty tableIdentifier or path.

    operationName

    Operation that is getting the DeltaLog, used in error messages.

    hadoopConf

    Hadoop file system options used to build DeltaLog.

    returns

    DeltaLog of the table

    Attributes
    protected
    Definition Classes
    DeltaCommand
    Exceptions thrown

    AnalysisException If either no Delta table exists at the given path/identifier or there is neither path nor tableIdentifier is provided.

  70. def getDeltaTable(target: LogicalPlan, cmd: String): DeltaTableV2

    Extracts the DeltaTableV2 from a LogicalPlan iff the LogicalPlan is a ResolvedTable with either a DeltaTableV2 or a V1Table that is referencing a Delta table.

    Extracts the DeltaTableV2 from a LogicalPlan iff the LogicalPlan is a ResolvedTable with either a DeltaTableV2 or a V1Table that is referencing a Delta table. In all other cases this method will throw a "Table not found" exception.

    Definition Classes
    DeltaCommand
  71. def getDeltaTablePathOrIdentifier(target: LogicalPlan, cmd: String): (Option[TableIdentifier], Option[String])

    Helper method to extract the table id or path from a LogicalPlan representing a Delta table.

    Helper method to extract the table id or path from a LogicalPlan representing a Delta table. This uses DeltaCommand.getDeltaTable to convert the LogicalPlan to a DeltaTableV2 and then extracts either the path or identifier from it. If the DeltaTableV2 has a CatalogTable, the table identifier will be returned. Otherwise, the table's path will be returned. Throws an exception if the LogicalPlan does not represent a Delta table.

    Definition Classes
    DeltaCommand
  72. def getMetadataAttributeByName(name: String): AttributeReference
    Definition Classes
    LogicalPlan
  73. def getMetadataAttributeByNameOpt(name: String): Option[AttributeReference]
    Definition Classes
    LogicalPlan
  74. def getSourceDF(): DataFrame
    Attributes
    protected
    Definition Classes
    MergeIntoMaterializeSource
  75. def getTableCatalogTable(target: LogicalPlan, cmd: String): Option[CatalogTable]

    Extracts CatalogTable metadata from a LogicalPlan if the plan is a ResolvedTable.

    Extracts CatalogTable metadata from a LogicalPlan if the plan is a ResolvedTable. The table can be a non delta table.

    Definition Classes
    DeltaCommand
  76. def getTablePathOrIdentifier(target: LogicalPlan, cmd: String): (Option[TableIdentifier], Option[String])

    Helper method to extract the table id or path from a LogicalPlan representing a resolved table or path.

    Helper method to extract the table id or path from a LogicalPlan representing a resolved table or path. This calls getDeltaTablePathOrIdentifier if the resolved table is a delta table. For non delta table with identifier, we extract its identifier. For non delta table with path, it expects the path to be wrapped in an ResolvedPathBasedNonDeltaTable and extracts it from there.

    Definition Classes
    DeltaCommand
  77. def getTagValue[T](tag: TreeNodeTag[T]): Option[T]
    Definition Classes
    TreeNode
  78. def getTargetOnlyPredicates(spark: SparkSession): Seq[Expression]
    Attributes
    protected
  79. def getTargetOutputCols(txn: OptimisticTransaction, makeNullable: Boolean = false): Seq[NamedExpression]

    Get the expression references for the output columns of the target table relative to the transaction.

    Get the expression references for the output columns of the target table relative to the transaction. Due to schema evolution, there are two kinds of expressions here: * References to columns in the target dataframe. Note that these references may have a different data type than they originally did due to schema evolution, but the exprId will be the same. These references will be marked as nullable if makeNullable is set to true. * Literal nulls, for new columns which are being added to the target table as part of this transaction, since new columns will have a value of null for all existing rows.

    Attributes
    protected
  80. def getTouchedFile(basePath: Path, filePath: String, nameToAddFileMap: Map[String, AddFile]): AddFile

    Find the AddFile record corresponding to the file that was read as part of a delete/update/merge operation.

    Find the AddFile record corresponding to the file that was read as part of a delete/update/merge operation.

    filePath

    The path to a file. Can be either absolute or relative

    nameToAddFileMap

    Map generated through generateCandidateFileMap()

    Definition Classes
    DeltaCommand
  81. def hasBeenExecuted(txn: OptimisticTransaction, sparkSession: SparkSession, options: Option[DeltaOptions] = None): Boolean

    Returns true if there is information in the spark session that indicates that this write has already been successfully written.

    Returns true if there is information in the spark session that indicates that this write has already been successfully written.

    Attributes
    protected
    Definition Classes
    DeltaCommand
  82. def hashCode(): Int
    Definition Classes
    TreeNode → AnyRef → Any
  83. def includesDeletes: Boolean

    Whether this merge statement includes delete statements.

    Whether this merge statement includes delete statements.

    Attributes
    protected
  84. def includesInserts: Boolean

    Whether this merge statement includes inserts statements.

    Whether this merge statement includes inserts statements.

    Attributes
    protected
  85. def incrementMetricAndReturnBool(name: String, valueToReturn: Boolean): Expression

    returns

    An Expression to increment a SQL metric

    Attributes
    protected
  86. def incrementMetricsAndReturnBool(names: Seq[String], valueToReturn: Boolean): Expression

    returns

    An Expression to increment SQL metrics

    Attributes
    protected
  87. def inferAdditionalConstraints(constraints: ExpressionSet): ExpressionSet
    Definition Classes
    ConstraintHelper
  88. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  89. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  90. def innerChildren: Seq[QueryPlan[_]]
    Definition Classes
    QueryPlan → TreeNode
  91. def inputSet: AttributeSet
    Definition Classes
    QueryPlan
  92. final def invalidateStatsCache(): Unit
    Definition Classes
    LogicalPlanStats
  93. def isCanonicalizedPlan: Boolean
    Attributes
    protected
    Definition Classes
    QueryPlan
  94. def isCatalogTable(analyzer: Analyzer, tableIdent: TableIdentifier): Boolean

    Use the analyzer to see whether the provided TableIdentifier is for a path based table or not

    Use the analyzer to see whether the provided TableIdentifier is for a path based table or not

    analyzer

    The session state analyzer to call

    tableIdent

    Table Identifier to determine whether is path based or not

    returns

    Boolean where true means that the table is a table in a metastore and false means the table is a path based table

    Definition Classes
    DeltaCommand
  95. def isCdcEnabled(deltaTxn: OptimisticTransaction): Boolean
    Attributes
    protected
  96. def isInsertOnly: Boolean

    Whether this merge statement only has only insert (NOT MATCHED) clauses.

    Whether this merge statement only has only insert (NOT MATCHED) clauses.

    Attributes
    protected
  97. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  98. def isLikelySelective(e: Expression): Boolean
    Definition Classes
    PredicateHelper
  99. def isMatchedOnly: Boolean

    Whether this merge statement has only MATCHED clauses.

    Whether this merge statement has only MATCHED clauses.

    Attributes
    protected
  100. def isNullIntolerant(expr: Expression): Boolean
    Attributes
    protected
    Definition Classes
    PredicateHelper
  101. val isOnlyOneUnconditionalDelete: Boolean
    Attributes
    protected
  102. def isPathIdentifier(tableIdent: TableIdentifier): Boolean

    Checks if the given identifier can be for a delta table's path

    Checks if the given identifier can be for a delta table's path

    tableIdent

    Table Identifier for which to check

    Attributes
    protected
    Definition Classes
    DeltaCommand
  103. def isRuleIneffective(ruleId: RuleId): Boolean
    Attributes
    protected
    Definition Classes
    TreeNode
  104. def isStreaming: Boolean
    Definition Classes
    LogicalPlan
  105. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  106. def jsonFields: List[JField]
    Attributes
    protected
    Definition Classes
    TreeNode
  107. final def legacyWithNewChildren(newChildren: Seq[LogicalPlan]): LogicalPlan
    Attributes
    protected
    Definition Classes
    TreeNode
  108. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  109. def logConsole(line: String): Unit
    Definition Classes
    DatabricksLogging
  110. def logDebug(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  111. def logDebug(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  112. def logError(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  113. def logError(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  114. def logInfo(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  115. def logInfo(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  116. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  117. def logTrace(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  118. def logTrace(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  119. def logWarning(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  120. def logWarning(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  121. def makeCopy(newArgs: Array[AnyRef]): LogicalPlan
    Definition Classes
    TreeNode
  122. def map[A](f: (LogicalPlan) => A): Seq[A]
    Definition Classes
    TreeNode
  123. final def mapChildren(f: (LogicalPlan) => LogicalPlan): LogicalPlan
    Definition Classes
    LeafLike
  124. def mapExpressions(f: (Expression) => Expression): MergeIntoCommandBase.this.type
    Definition Classes
    QueryPlan
  125. def mapProductIterator[B](f: (Any) => B)(implicit arg0: ClassTag[B]): Array[B]
    Attributes
    protected
    Definition Classes
    TreeNode
  126. def markRuleAsIneffective(ruleId: RuleId): Unit
    Attributes
    protected
    Definition Classes
    TreeNode
  127. val materializedSourceRDD: Option[RDD[InternalRow]]

    If the source was materialized, reference to the checkpointed RDD.

    If the source was materialized, reference to the checkpointed RDD.

    Attributes
    protected
    Definition Classes
    MergeIntoMaterializeSource
  128. def maxRows: Option[Long]
    Definition Classes
    LogicalPlan
  129. def maxRowsPerPartition: Option[Long]
    Definition Classes
    LogicalPlan
  130. def metadataOutput: Seq[Attribute]
    Definition Classes
    LogicalPlan
  131. lazy val metrics: Map[String, SQLMetric]
    Definition Classes
    MergeIntoCommandBase → RunnableCommand
  132. final def missingInput: AttributeSet
    Definition Classes
    QueryPlan
  133. def multiTransformDown(rule: PartialFunction[LogicalPlan, Seq[LogicalPlan]]): Stream[LogicalPlan]
    Definition Classes
    TreeNode
  134. def multiTransformDownWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[LogicalPlan, Seq[LogicalPlan]]): Stream[LogicalPlan]
    Definition Classes
    TreeNode
  135. var multipleMatchDeleteOnlyOvercount: Option[Long]
    Attributes
    protected
  136. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  137. def nodeName: String
    Definition Classes
    TreeNode
  138. final val nodePatterns: Seq[TreePattern]
    Definition Classes
    Command → TreeNode
  139. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  140. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  141. def numberedTreeString: String
    Definition Classes
    TreeNode
  142. val origin: Origin
    Definition Classes
    TreeNode → WithOrigin
  143. def otherCopyArgs: Seq[AnyRef]
    Attributes
    protected
    Definition Classes
    TreeNode
  144. def output: Seq[Attribute]
    Definition Classes
    Command → QueryPlan
  145. def outputOrdering: Seq[SortOrder]
    Definition Classes
    QueryPlan
  146. lazy val outputSet: AttributeSet
    Definition Classes
    QueryPlan
    Annotations
    @transient()
  147. def outputWithNullability(output: Seq[Attribute], nonNullAttrExprIds: Seq[ExprId]): Seq[Attribute]
    Attributes
    protected
    Definition Classes
    PredicateHelper
  148. def p(number: Int): LogicalPlan
    Definition Classes
    TreeNode
  149. def parsePredicates(spark: SparkSession, predicate: String): Seq[Expression]

    Converts string predicates into Expressions relative to a transaction.

    Converts string predicates into Expressions relative to a transaction.

    Attributes
    protected
    Definition Classes
    DeltaCommand
    Exceptions thrown

    AnalysisException if a non-partition column is referenced.

  150. def planContainsOnlyDeltaScans(source: LogicalPlan): Boolean
    Attributes
    protected
    Definition Classes
    DeltaSparkPlanUtils
  151. def planIsDeterministic(plan: LogicalPlan): Boolean

    Returns true if plan has a safe level of determinism.

    Returns true if plan has a safe level of determinism. This is a conservative approximation of plan being a truly deterministic query.

    Attributes
    protected
    Definition Classes
    DeltaSparkPlanUtils
  152. def prepareSourceDFAndReturnMaterializeReason(spark: SparkSession, source: LogicalPlan, condition: Expression, matchedClauses: Seq[DeltaMergeIntoMatchedClause], notMatchedClauses: Seq[DeltaMergeIntoNotMatchedClause], isInsertOnly: Boolean): MergeIntoMaterializeSourceReason

    If source needs to be materialized, prepare the materialized dataframe in sourceDF Otherwise, prepare regular dataframe.

    If source needs to be materialized, prepare the materialized dataframe in sourceDF Otherwise, prepare regular dataframe.

    returns

    the source materialization reason

    Attributes
    protected
    Definition Classes
    MergeIntoMaterializeSource
  153. def prettyJson: String
    Definition Classes
    TreeNode
  154. def printSchema(): Unit
    Definition Classes
    QueryPlan
  155. def producedAttributes: AttributeSet
    Definition Classes
    Command → QueryPlan
  156. def productElementName(n: Int): String
    Definition Classes
    Product
  157. def productElementNames: Iterator[String]
    Definition Classes
    Product
  158. def productIterator: Iterator[Any]
    Definition Classes
    Product
  159. def productPrefix: String
    Definition Classes
    Product
  160. def recordDeltaEvent(deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty, data: AnyRef = null, path: Option[Path] = None): Unit

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    path

    Used to log the path of the delta table when deltaLog is null.

    Attributes
    protected
    Definition Classes
    DeltaLogging
  161. def recordDeltaOperation[A](deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: => A): A

    Used to report the duration as well as the success or failure of an operation on a deltaLog.

    Used to report the duration as well as the success or failure of an operation on a deltaLog.

    Attributes
    protected
    Definition Classes
    DeltaLogging
  162. def recordDeltaOperationForTablePath[A](tablePath: String, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: => A): A

    Used to report the duration as well as the success or failure of an operation on a tahoePath.

    Used to report the duration as well as the success or failure of an operation on a tahoePath.

    Attributes
    protected
    Definition Classes
    DeltaLogging
  163. def recordEvent(metric: MetricDefinition, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit
    Definition Classes
    DatabricksLogging
  164. def recordFrameProfile[T](group: String, name: String)(thunk: => T): T
    Attributes
    protected
    Definition Classes
    DeltaLogging
  165. def recordMergeOperation[A](extraOpType: String = "", status: String = null, sqlMetricName: String = null)(thunk: => A): A

    Execute the given thunk and return its result while recording the time taken to do it and setting additional local properties for better UI visibility.

    Execute the given thunk and return its result while recording the time taken to do it and setting additional local properties for better UI visibility.

    extraOpType

    extra operation name recorded in the logs

    status

    human readable status string describing what the thunk is doing

    sqlMetricName

    name of SQL metric to update with the time taken by the thunk

    thunk

    the code to execute

    Attributes
    protected
  166. def recordOperation[S](opType: OpType, opTarget: String = null, extraTags: Map[TagDefinition, String], isSynchronous: Boolean = true, alwaysRecordStats: Boolean = false, allowAuthTags: Boolean = false, killJvmIfStuck: Boolean = false, outputMetric: MetricDefinition = METRIC_OPERATION_DURATION, silent: Boolean = true)(thunk: => S): S
    Definition Classes
    DatabricksLogging
  167. def recordProductEvent(metric: MetricDefinition with CentralizableMetric, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit
    Definition Classes
    DatabricksLogging
  168. def recordProductUsage(metric: MetricDefinition with CentralizableMetric, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit
    Definition Classes
    DatabricksLogging
  169. def recordUsage(metric: MetricDefinition, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit
    Definition Classes
    DatabricksLogging
  170. lazy val references: AttributeSet
    Definition Classes
    QueryPlan
    Annotations
    @transient()
  171. def refresh(): Unit
    Definition Classes
    LogicalPlan
  172. def removeFilesFromPaths(deltaLog: DeltaLog, nameToAddFileMap: Map[String, AddFile], filesToRewrite: Seq[String], operationTimestamp: Long): Seq[RemoveFile]

    This method provides the RemoveFile actions that are necessary for files that are touched and need to be rewritten in methods like Delete, Update, and Merge.

    This method provides the RemoveFile actions that are necessary for files that are touched and need to be rewritten in methods like Delete, Update, and Merge.

    deltaLog

    The DeltaLog of the table that is being operated on

    nameToAddFileMap

    A map generated using generateCandidateFileMap.

    filesToRewrite

    Absolute paths of the files that were touched. We will search for these in candidateFiles. Obtained as the output of the input_file_name function.

    operationTimestamp

    The timestamp of the operation

    Attributes
    protected
    Definition Classes
    DeltaCommand
  173. def replaceAlias(expr: Expression, aliasMap: AttributeMap[Alias]): Expression
    Attributes
    protected
    Definition Classes
    AliasHelper
  174. def replaceAliasButKeepName(expr: NamedExpression, aliasMap: AttributeMap[Alias]): NamedExpression
    Attributes
    protected
    Definition Classes
    AliasHelper
  175. def resolve(nameParts: Seq[String], resolver: Resolver): Option[NamedExpression]
    Definition Classes
    LogicalPlan
  176. def resolve(schema: StructType, resolver: Resolver): Seq[Attribute]
    Definition Classes
    LogicalPlan
  177. def resolveChildren(nameParts: Seq[String], resolver: Resolver): Option[NamedExpression]
    Definition Classes
    LogicalPlan
  178. def resolveExpressions(r: PartialFunction[Expression, Expression]): LogicalPlan
    Definition Classes
    AnalysisHelper
  179. def resolveExpressionsWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[Expression, Expression]): LogicalPlan
    Definition Classes
    AnalysisHelper
  180. def resolveIdentifier(analyzer: Analyzer, identifier: TableIdentifier): LogicalPlan

    Use the analyzer to resolve the identifier provided

    Use the analyzer to resolve the identifier provided

    analyzer

    The session state analyzer to call

    identifier

    Table Identifier to determine whether is path based or not

    Attributes
    protected
    Definition Classes
    DeltaCommand
  181. def resolveOperators(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    AnalysisHelper
  182. def resolveOperatorsDown(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    AnalysisHelper
  183. def resolveOperatorsDownWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    AnalysisHelper
  184. def resolveOperatorsUp(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    AnalysisHelper
  185. def resolveOperatorsUpWithNewOutput(rule: PartialFunction[LogicalPlan, (LogicalPlan, Seq[(Attribute, Attribute)])]): LogicalPlan
    Definition Classes
    AnalysisHelper
  186. def resolveOperatorsUpWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    AnalysisHelper
  187. def resolveOperatorsWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    AnalysisHelper
  188. def resolveQuoted(name: String, resolver: Resolver): Option[NamedExpression]
    Definition Classes
    LogicalPlan
  189. lazy val resolved: Boolean
    Definition Classes
    LogicalPlan
  190. def rewriteAttrs(attrMap: AttributeMap[Attribute]): LogicalPlan
    Definition Classes
    QueryPlan
  191. def run(spark: SparkSession): Seq[Row]
    Definition Classes
    MergeIntoCommandBase → RunnableCommand
  192. def runWithMaterializedSourceLostRetries(spark: SparkSession, deltaLog: DeltaLog, metrics: Map[String, SQLMetric], runMergeFunc: (SparkSession) => Seq[Row]): Seq[Row]

    Run the Merge with retries in case it detects an RDD block lost error of the materialized source RDD.

    Run the Merge with retries in case it detects an RDD block lost error of the materialized source RDD. It will also record out of disk error, if such happen - possibly because of increased disk pressure from the materialized source RDD.

    Attributes
    protected
    Definition Classes
    MergeIntoMaterializeSource
  193. def sameOutput(other: LogicalPlan): Boolean
    Definition Classes
    LogicalPlan
  194. final def sameResult(other: LogicalPlan): Boolean
    Definition Classes
    QueryPlan
  195. lazy val sc: SparkContext
    Attributes
    protected
    Annotations
    @transient()
  196. lazy val schema: StructType
    Definition Classes
    QueryPlan
  197. def schemaString: String
    Definition Classes
    QueryPlan
  198. final def semanticHash(): Int
    Definition Classes
    QueryPlan
  199. def sendDriverMetrics(spark: SparkSession, metrics: Map[String, SQLMetric]): Unit

    Send the driver-side metrics.

    Send the driver-side metrics.

    This is needed to make the SQL metrics visible in the Spark UI. All metrics are default initialized with 0 so that's what we're reporting in case we skip an already executed action.

    Attributes
    protected
    Definition Classes
    DeltaCommand
  200. def seqToString(exprs: Seq[Expression]): String
    Attributes
    protected
  201. def setTagValue[T](tag: TreeNodeTag[T], value: T): Unit
    Definition Classes
    TreeNode
  202. def shouldMaterializeSource(spark: SparkSession, source: LogicalPlan, isInsertOnly: Boolean): (Boolean, MergeIntoMaterializeSourceReason)

    returns

    pair of boolean whether source should be materialized and the source materialization reason

    Attributes
    protected
    Definition Classes
    MergeIntoMaterializeSource
  203. def shouldOptimizeMatchedOnlyMerge(spark: SparkSession): Boolean
    Attributes
    protected
  204. def simpleString(maxFields: Int): String
    Definition Classes
    QueryPlan → TreeNode
  205. def simpleStringWithNodeId(): String
    Definition Classes
    QueryPlan → TreeNode
  206. def splitConjunctivePredicates(condition: Expression): Seq[Expression]
    Attributes
    protected
    Definition Classes
    PredicateHelper
  207. def splitDisjunctivePredicates(condition: Expression): Seq[Expression]
    Attributes
    protected
    Definition Classes
    PredicateHelper
  208. def statePrefix: String
    Attributes
    protected
    Definition Classes
    LogicalPlan → QueryPlan
  209. def stats: Statistics
    Definition Classes
    Command → LogicalPlanStats
  210. val statsCache: Option[Statistics]
    Attributes
    protected
    Definition Classes
    LogicalPlanStats
  211. def stringArgs: Iterator[Any]
    Attributes
    protected
    Definition Classes
    TreeNode
  212. lazy val subqueries: Seq[LogicalPlan]
    Definition Classes
    QueryPlan
    Annotations
    @transient()
  213. def subqueriesAll: Seq[LogicalPlan]
    Definition Classes
    QueryPlan
  214. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  215. lazy val targetDeltaLog: DeltaLog
    Attributes
    protected
    Annotations
    @transient()
  216. def throwErrorOnMultipleMatches(hasMultipleMatches: Boolean, spark: SparkSession): Unit
    Attributes
    protected
  217. def toJSON: String
    Definition Classes
    TreeNode
  218. def toString(): String
    Definition Classes
    TreeNode → AnyRef → Any
  219. def transform(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    TreeNode
  220. def transformAllExpressions(rule: PartialFunction[Expression, Expression]): MergeIntoCommandBase.this.type
    Definition Classes
    QueryPlan
  221. def transformAllExpressionsWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[Expression, Expression]): MergeIntoCommandBase.this.type
    Definition Classes
    AnalysisHelper → QueryPlan
  222. def transformAllExpressionsWithSubqueries(rule: PartialFunction[Expression, Expression]): MergeIntoCommandBase.this.type
    Definition Classes
    QueryPlan
  223. def transformDown(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    TreeNode
  224. def transformDownWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    AnalysisHelper → TreeNode
  225. def transformDownWithSubqueries(f: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    QueryPlan
  226. def transformDownWithSubqueriesAndPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(f: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    QueryPlan
  227. def transformExpressions(rule: PartialFunction[Expression, Expression]): MergeIntoCommandBase.this.type
    Definition Classes
    QueryPlan
  228. def transformExpressionsDown(rule: PartialFunction[Expression, Expression]): MergeIntoCommandBase.this.type
    Definition Classes
    QueryPlan
  229. def transformExpressionsDownWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[Expression, Expression]): MergeIntoCommandBase.this.type
    Definition Classes
    QueryPlan
  230. def transformExpressionsUp(rule: PartialFunction[Expression, Expression]): MergeIntoCommandBase.this.type
    Definition Classes
    QueryPlan
  231. def transformExpressionsUpWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[Expression, Expression]): MergeIntoCommandBase.this.type
    Definition Classes
    QueryPlan
  232. def transformExpressionsWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[Expression, Expression]): MergeIntoCommandBase.this.type
    Definition Classes
    QueryPlan
  233. def transformUp(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    TreeNode
  234. def transformUpWithBeforeAndAfterRuleOnChildren(cond: (LogicalPlan) => Boolean, ruleId: RuleId)(rule: PartialFunction[(LogicalPlan, LogicalPlan), LogicalPlan]): LogicalPlan
    Definition Classes
    TreeNode
  235. def transformUpWithNewOutput(rule: PartialFunction[LogicalPlan, (LogicalPlan, Seq[(Attribute, Attribute)])], skipCond: (LogicalPlan) => Boolean, canGetOutput: (LogicalPlan) => Boolean): LogicalPlan
    Definition Classes
    AnalysisHelper → QueryPlan
  236. def transformUpWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    AnalysisHelper → TreeNode
  237. def transformUpWithSubqueries(f: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    QueryPlan
  238. def transformWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    TreeNode
  239. def transformWithSubqueries(f: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan
    Definition Classes
    QueryPlan
  240. lazy val treePatternBits: BitSet
    Definition Classes
    QueryPlan → TreeNode → TreePatternBits
  241. def treeString(append: (String) => Unit, verbose: Boolean, addSuffix: Boolean, maxFields: Int, printOperatorId: Boolean): Unit
    Definition Classes
    TreeNode
  242. final def treeString(verbose: Boolean, addSuffix: Boolean, maxFields: Int, printOperatorId: Boolean): String
    Definition Classes
    TreeNode
  243. final def treeString: String
    Definition Classes
    TreeNode
  244. def trimAliases(e: Expression): Expression
    Attributes
    protected
    Definition Classes
    AliasHelper
  245. def trimNonTopLevelAliases[T <: Expression](e: T): T
    Attributes
    protected
    Definition Classes
    AliasHelper
  246. def unsetTagValue[T](tag: TreeNodeTag[T]): Unit
    Definition Classes
    TreeNode
  247. final def updateMetadata(spark: SparkSession, txn: OptimisticTransaction, schema: StructType, partitionColumns: Seq[String], configuration: Map[String, String], isOverwriteMode: Boolean, rearrangeOnly: Boolean): Unit
    Attributes
    protected
    Definition Classes
    ImplicitMetadataOperation
  248. def updateOuterReferencesInSubquery(plan: LogicalPlan, attrMap: AttributeMap[Attribute]): LogicalPlan
    Definition Classes
    AnalysisHelper → QueryPlan
  249. lazy val validConstraints: ExpressionSet
    Attributes
    protected
    Definition Classes
    QueryPlanConstraints
  250. def verboseString(maxFields: Int): String
    Definition Classes
    QueryPlan → TreeNode
  251. def verboseStringWithOperatorId(): String
    Definition Classes
    QueryPlan
  252. def verboseStringWithSuffix(maxFields: Int): String
    Definition Classes
    LogicalPlan → TreeNode
  253. def verifyPartitionPredicates(spark: SparkSession, partitionColumns: Seq[String], predicates: Seq[Expression]): Unit
    Definition Classes
    DeltaCommand
  254. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  255. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  256. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  257. final def withNewChildren(newChildren: Seq[LogicalPlan]): LogicalPlan
    Definition Classes
    TreeNode
  258. def withNewChildrenInternal(newChildren: IndexedSeq[LogicalPlan]): LogicalPlan
    Definition Classes
    LeafLike
  259. def withStatusCode[T](statusCode: String, defaultMessage: String, data: Map[String, Any] = Map.empty)(body: => T): T

    Report a log to indicate some command is running.

    Report a log to indicate some command is running.

    Definition Classes
    DeltaProgressReporter
  260. def writeFiles(spark: SparkSession, txn: OptimisticTransaction, outputDF: DataFrame): Seq[FileAction]

    Write the output data to files, repartitioning the output DataFrame by the partition columns if table is partitioned and merge.repartitionBeforeWrite.enabled is set to true.

    Write the output data to files, repartitioning the output DataFrame by the partition columns if table is partitioned and merge.repartitionBeforeWrite.enabled is set to true.

    Attributes
    protected
  261. object RetryHandling extends Enumeration
    Definition Classes
    MergeIntoMaterializeSource
  262. object SubqueryExpression

    Extractor object for the subquery plan of expressions that contain subqueries.

    Extractor object for the subquery plan of expressions that contain subqueries.

    Definition Classes
    DeltaSparkPlanUtils

Inherited from MergeIntoMaterializeSource

Inherited from DeltaSparkPlanUtils

Inherited from ImplicitMetadataOperation

Inherited from PredicateHelper

Inherited from AliasHelper

Inherited from DeltaCommand

Inherited from DeltaLogging

Inherited from DatabricksLogging

Inherited from DeltaProgressReporter

Inherited from LeafRunnableCommand

Inherited from LeafLike[LogicalPlan]

Inherited from RunnableCommand

Inherited from Command

Inherited from LogicalPlan

Inherited from Logging

Inherited from QueryPlanConstraints

Inherited from ConstraintHelper

Inherited from LogicalPlanDistinctKeys

Inherited from LogicalPlanStats

Inherited from AnalysisHelper

Inherited from QueryPlan[LogicalPlan]

Inherited from SQLConfHelper

Inherited from TreeNode[LogicalPlan]

Inherited from WithOrigin

Inherited from TreePatternBits

Inherited from Product

Inherited from Equals

Inherited from AnyRef

Inherited from Any

Ungrouped