Returns whether it allows non-deterministic expressions.

Track which attempt or retry it is in runWithMaterializedSourceAndRetries

Build a base relation of files that need to be rewritten as part of an update/delete/merge operation.

Builds a new logical plan to read the given files instead of the whole target table. The plan returned has the same output columns (exprIds) as the target logical plan, so that existing update/insert expressions can be applied on this new plan. Unneeded non-partition columns may be dropped.

Builds a new logical plan to read the target table using the given fileIndex. The plan returned has the same output columns (exprIds) as the target logical plan, so that existing update/insert expressions can be applied on this new plan.

Add a cast to the child expression if it differs from the specified data type. Note that structs here are cast by name, rather than the Spark SQL default of casting by position.

Evaluate that invariant "holds" (evaluates to true) or log the violation.

Verify that the high water marks used by the identity column generators still match the the high water marks in the version of the table read by the current transaction. These high water marks were determined during analysis in PreprocessTableMerge, which runs outside of the current transaction, so they may no longer be valid.

Throws an exception if merge metrics indicate that the source table changed between the first and the second source table scans.

Collects the merge operation stats and metrics into a MergeStats object that can be recorded with recordDeltaEvent. Merge stats should be collected after committing all new actions as metrics may still be updated during commit.

Returns whether an expression contains any deterministic UDFs.

Returns whether the read predicates of a transaction contain any deterministic UDFs.

Returns SetTransaction if a valid app ID and version are present. Otherwise returns an empty list.

Helper method to check invariants in Delta code. Fails when running in tests, records a delta assertion event and logs a warning otherwise.

Returns a part of the plan that does not have a safe level of determinism. This is a conservative approximation of plan being a truly deterministic query.

Generates a map of file names to add file entries for operations where we will need to rewrite files such as delete, merge, update. We expect file names to be unique, because each file contains a UUID.

See docs on overloaded method.

Given a target schema and a set of update operations, generate a list of update expressions, which are aligned with the given schema.

For update operations to nested struct fields, this method recursively walks down schema tree and apply the update expressions along the way. For example, assume table target has the following schema: s1 struct<a: int, b: int, c: int>, s2 struct<a: int, b: int>, z int

Given an update command:

• UPDATE target SET s1.a = 1, s1.b = 2, z = 3

this method works as follows:

generateUpdateExpressions( targetSchema=[s1,s2,z], defaultExprs=[s1,s2, z], updateOps=[(s1.a, 1), (s1.b, 2), (z, 3)]) -> generates expression for s1 - build recursively from child assignments generateUpdateExpressions( targetSchema=[a,b,c], defaultExprs=[a, b, c], updateOps=[(a, 1),(b, 2)], pathPrefix=["s1"]) end-of-recursion -> returns (1, 2, a.c) -> generates expression for s2 - no child assignment and no update expression: use default expression s2 -> generates expression for z - use available update expression 3 -> returns ((1, 2, a.c), s2, 3)

Generate update expressions for generated columns that the user doesn't provide a update expression. For each item in updateExprs that's None, we will find its generation expression from generatedColumns. In order to resolve this generation expression, we will create a fake Project which contains all update expressions and resolve the generation expression with this project. Source columns of a generation expression will also be replaced with their corresponding update expressions.

For example, given a table that has a generated column g defined as c1 + 10. For the following update command:

UPDATE target SET c1 = c2 + 100, c2 = 1000

We will generate the update expression (c2 + 100) + 10 for column g. Note: in this update expression, we should use the old c2 attribute rather than its new value 1000.

Utility method to return the DeltaLog of an existing Delta table referred by either the given path or tableIdentifier.

Extracts the DeltaTableV2 from a LogicalPlan iff the LogicalPlan is a ResolvedTable with either a DeltaTableV2 or a V1Table that is referencing a Delta table. In all other cases this method will throw a "Table not found" exception.

Helper method to extract the table id or path from a LogicalPlan representing a Delta table. This uses DeltaCommand.getDeltaTable to convert the LogicalPlan to a DeltaTableV2 and then extracts either the path or identifier from it. If the DeltaTableV2 has a CatalogTable, the table identifier will be returned. Otherwise, the table's path will be returned. Throws an exception if the LogicalPlan does not represent a Delta table.

Returns the prepared merge source.

Returns a sequence of new DomainMetadata if canUpdateMetadata is true and the operation is either create table or replace the whole table (not replaceWhere operation). This is because we only update Domain Metadata when creating or replacing table, and replace table for DDL and DataFrameWriterV2 are already handled in CreateDeltaTableCommand. In that case, canUpdateMetadata is false, so we don't update again.

Extracts CatalogTable metadata from a LogicalPlan if the plan is a ResolvedTable. The table can be a non delta table.

Helper method to extract the table id or path from a LogicalPlan representing a resolved table or path. This calls getDeltaTablePathOrIdentifier if the resolved table is a delta table. For non delta table with identifier, we extract its identifier. For non delta table with path, it expects the path to be wrapped in an ResolvedPathBasedNonDeltaTable and extracts it from there.

Find the AddFile record corresponding to the file that was read as part of a delete/update/merge operation.

Returns true if there is information in the spark session that indicates that this write has already been successfully written.

Whether this merge statement includes delete statements.

Whether this merge statement includes inserts statements.

Use the analyzer to see whether the provided TableIdentifier is for a path based table or not

Whether this merge statement only has delete clauses.

Whether this merge statement only has insert (NOT MATCHED) clauses.

Whether this merge statement only has MATCHED clauses.

Checks if the given identifier can be for a delta table's path

If the source was materialized, reference to the checkpointed RDD.

Converts string predicates into Expressions relative to a transaction.

Returns true if plan has a safe level of determinism. This is a conservative approximation of plan being a truly deterministic query.

Expressions to convert from a pre-evolution target row to the post-evolution target row. These expressions are used for columns that are not modified in updated rows or to copy rows that are not modified. There are two kinds of expressions here: * References to existing columns in the target dataframe. Note that these references may have a different data type than they originally did due to schema evolution so we add a cast that supports schema evolution. The references will be marked as nullable if makeNullable is set to true, which allows the attributes to reference the output of an outer join. * Literal nulls, for new columns which are being added to the target table as part of this transaction, since new columns will have a value of null for all existing rows.

If source needs to be materialized, prepare the materialized dataframe in sourceDF Otherwise, prepare regular dataframe.

Used to record the occurrence of a single event or report detailed, operation specific statistics.

Used to report the duration as well as the success or failure of an operation on a deltaLog.

Used to report the duration as well as the success or failure of an operation on a tahoePath.

Execute the given thunk and return its result while recording the time taken to do it and setting additional local properties for better UI visibility.

This method provides the RemoveFile actions that are necessary for files that are touched and need to be rewritten in methods like Delete, Update, and Merge.

Use the analyzer to resolve the identifier provided

Resolve expressions using the attributes provided by planProvidingAttrs. Throw an error if failing to resolve any expressions.

Run the Merge with retries in case it detects an RDD block lost error of the materialized source RDD. It will also record out of disk error, if such happens - possibly because of increased disk pressure from the materialized source RDD.

Send the driver-side metrics.

This is needed to make the SQL metrics visible in the Spark UI. All metrics are default initialized with 0 so that's what we're reporting in case we skip an already executed action.

Sets operationNumSourceRows as numSourceRowsInSecondScan if the source was scanned without possibility of pruning in the 2nd scan. Uses numSourceRows otherwise.