The path to the file that holds metadata about the most recent checkpoint.

Helper trait to instantiate the icebergConverter member variable of the DeltaLog. We do this through reflection so that delta-spark doesn't have a compile-time dependency on the shaded iceberg module.

Asserts that the table's protocol enabled all features that are active in the metadata.

A mismatch shouldn't happen when the table has gone through a proper write process because we require all active features during writes. However, other clients may void this guarantee.

Verify the required Spark conf for delta Throw DeltaErrors.configureSparkSessionWithExtensionAndCatalog exception if spark.sql.catalog.spark_catalog config is missing. We do not check for spark.sql.extensions because DeltaSparkSessionExtension can alternatively be activated using the .withExtension() API. This check can be disabled by setting DELTA_CHECK_REQUIRED_SPARK_CONF to false.

Creates a checkpoint using snapshotToCheckpoint. By default it uses the current log version. Note that this function captures and logs all exceptions, since the checkpoint shouldn't fail the overall commit operation.

Returns the checkpoint interval for this log. Not transactional.

Create the log directory. Unlike ensureLogDirectoryExist, this method doesn't check whether the log directory exists and it will ignore the return value of mkdirs.

Returns a BaseRelation that contains all of the data present in the table. This relation will be continually updated as files are added or removed from the table. However, new BaseRelation must be requested in order to see changes to the schema.

Helper method to create a compatibility classic single file checkpoint file for this table. This is needed so that any legacy reader which do not understand V2CheckpointTableFeature could read the legacy classic checkpoint file and fail gracefully with Protocol requirement failure.

Creates a snapshot for a new delta commit.

Create a Snapshot from the given LogSegment. If failing to create the snapshot, we will search an equivalent LogSegment using a different checkpoint and retry up to DeltaSQLConf.DELTA_SNAPSHOT_LOADING_MAX_RETRIES times.

Use ReentrantLock to allow us to call lockInterruptibly

Returns the duration in millis for how long to keep around obsolete logs. We may keep logs beyond this duration until the next calendar day to avoid constantly creating checkpoints.

Whether to clean up expired log files and checkpoints.

Build the underlying Spark FileFormat of the Delta table with specified metadata.

With column mapping, some properties of the underlying file format might change during transaction, so if possible, we should always pass in the latest transaction's metadata instead of one from a past snapshot.

Finds a checkpoint such that we are able to construct table snapshot for all versions at or greater than the checkpoint version returned.

Finds the first verified, complete checkpoint before the given CheckpointInstance. If checkpointInstance is passed as None, then we return the last complete checkpoint in the deltalog directory.

Finds the first verified, complete checkpoint before the given version. Note that the returned checkpoint will always be < version.

Get access to all actions starting from "startVersion" (inclusive) via FileStatus. If startVersion doesn't exist, return an empty Iterator.

Get all actions starting from "startVersion" (inclusive). If startVersion doesn't exist, return an empty Iterator.

Returns the last known checkpoint version based on LastCheckpointInfo or CheckpointProvider. Returns -1 if both the info is not available.

Given a list of checkpoint files, pick the latest complete checkpoint instance which is not later than notLaterThan.

Used to compute the LogSegment after a commit

Helper function for the getLogSegmentForVersion above. Called with a provided files list, and will then try to construct a new LogSegment using that.

Get a list of files that can be used to compute a Snapshot at version versionToLoad, If versionToLoad is not provided, will generate the list of files that are needed to load the latest version of the Delta table. This method also performs checks to ensure that the delta files are contiguous.

Get the LogSegment that will help in computing the Snapshot of the table at DeltaLog initialization, or None if the directory was empty/missing.

We accept keys both with and without the spark. prefix to maintain compatibility across the Delta ecosystem

Get the snapshot at version.

Load the Snapshot for this Delta table at initialization. This method uses the lastCheckpoint file as a hint on where to start listing the transaction log directory. If the _delta_log directory doesn't exist, this method will return an InitialSnapshot.

Deletes any unreferenced files from the sidecar directory _delta_log/_sidecar

Creates a LogicalRelation for a given DeltaLogFileIndex, with all necessary file source options taken from the Delta Log. All reads of Delta metadata files should use this method.

Install the provided segmentOpt as the currentSnapshot on the cluster

Checks if the given timestamp is outside the current staleness window

Returns true if the path is delta log files. Delta log files can be delta commit file (e.g., 000000000.json), or checkpoint file. (e.g., 000000001.checkpoint.00001.00003.parquet)

Returns the delta files and checkpoint files starting from the given startVersion. versionToLoad is an optional parameter to set the max bound. It's usually used to load a table snapshot for a specific version.

Returns an iterator containing a list of files found from the provided path

Load the data using the FileIndex. This allows us to skip many checks that add overhead, e.g. file existence checks, partitioning schema inference.

Loads the given checkpoint manually to come up with the LastCheckpointInfo

The max lineage length of a Snapshot before Delta forces to build a Snapshot from scratch. Delta will build a Snapshot on top of the previous one if it doesn't see a checkpoint. However, there is a race condition that when two writers are writing at the same time, a writer may fail to pick up checkpoints written by another one, and the lineage will grow and finally cause StackOverflowError. Hence we have to force to build a Snapshot from scratch when the lineage length is too large to avoid hitting StackOverflowError.

Returns the Hadoop Configuration object which can be used to access the file system. All Delta code should use this method to create the Hadoop Configuration object, so that the hadoop file system configurations specified in DataFrame options will come into effect.

Used to record the occurrence of a single event or report detailed, operation specific statistics.

Used to report the duration as well as the success or failure of an operation on a deltaLog.

Used to report the duration as well as the success or failure of an operation on a tahoePath.

Replace the given snapshot with the provided one.

Path to sidecar directory. This is intentionally kept lazy val as otherwise any other constructor codepaths in DeltaLog (e.g. SnapshotManagement etc) will see it as null as they are executed before this line is called.

Return the current Spark session used.

Returns a new OptimisticTransaction that can be used to read the current state of the log and then commit updates. The reads and updates will be checked for logical conflicts with any concurrent writes to the log, and post-commit hooks can be used to notify the table's catalog of schema changes, etc.

Note that all reads in a transaction must go through the returned transaction object, and not directly to the DeltaLog otherwise they will not be checked for conflicts.

Used to read and write physical log files and checkpoints.

Whether a Delta table exists at this directory. It is okay to use the cached volatile snapshot here, since the worst case is that the table has recently started existing which hasn't been picked up here. If so, any subsequent command that updates the table will see the right value.

Returns the current snapshot. This does not automatically update().

WARNING: This is not guaranteed to give you the latest snapshot of the log, nor stay consistent across multiple accesses. If you need the latest snapshot, it is recommended to fetch it using deltaLog.update(); and save the returned snapshot so it does not unexpectedly change from under you. See how OptimisticTransaction and DeltaScan use the snapshot as examples for write/read paths respectively. This API should only be used in scenarios where any recent snapshot will suffice and an update is undesired, or by internal code that holds the DeltaLog lock to prevent races.

Update ActionLog by applying the new delta files if any.

Called after committing a transaction and updating the state of the table.

Queries the store for new delta files and applies them to the current state. Note: the caller should hold deltaLogLock before calling this method.

Upgrade the table's protocol version, by default to the maximum recognized reader and writer versions in this Delta release. This method only upgrades protocol version, and will fail if the new protocol version is not a superset of the original one used by the snapshot.

Check for conflicting LogStore configs in the spark configuration.

To maintain compatibility across the Delta ecosystem, we accept keys both with and without the "spark." prefix. This means for setting the class conf, we accept both "spark.delta.logStore.class" and "delta.logStore.class" and for scheme confs we accept both "spark.delta.logStore.${scheme}.impl" and "delta.logStore.${scheme}.impl"

If a conf is set both with and without the spark prefix, it must be set to the same value, otherwise we throw an error.

Catch non-fatal exceptions related to checkpointing, since the checkpoint is written after the commit has completed. From the perspective of the user, the commit has completed successfully. However, throw if this is in a testing environment - that way any breaking changes can be caught in unit tests.

Execute a piece of code within a new OptimisticTransaction. Reads/write sets will be recorded for this table, and all other tables will be read at a snapshot that is pinned on the first access.

Report a log to indicate some command is running.

Creates a checkpoint using the default snapshot.

WARNING: This API is being deprecated, and will be removed in future versions. Please use the checkpoint(Snapshot) function below to write checkpoints to the delta log.

WARNING: This API is unsafe and deprecated. It will be removed in future versions. Use the above unsafeVolatileSnapshot to get the most recently cached snapshot on the cluster.

Legacy/compat overload that does not require catalog table information. Avoid prod use.

Legacy/compat overload that does not require catalog table information. Avoid prod use.