Package io.delta.storage.commit.uccommitcoordinator

Class UCCommitCoordinatorClient

Object

io.delta.storage.commit.uccommitcoordinator.UCCommitCoordinatorClient

All Implemented Interfaces:

CommitCoordinatorClient

public class UCCommitCoordinatorClient
extends Object
implements CommitCoordinatorClient

A commit coordinator client that uses unity-catalog as the commit coordinator.

Field Summary

Fields

Modifier and Type	Field	Description
static int	BACKFILL_LISTING_OFFSET	Offset from current commit version for backfill listing optimization.
final Map<String,String>	conf	Configuration map containing settings for the coordinator client.
static final String	UC_METASTORE_ID_KEY	Key for identifying Unity Catalog metastore ID in `delta.coordinatedCommits.commitCoordinatorConf{-preview}`.
static final String	UC_TABLE_ID_KEY	Key for identifying Unity Catalog table ID in `delta.coordinatedCommits.tableConf{-preview}`.
static final String	UC_TABLE_ID_KEY_OLD
final UCClient	ucClient	Unity Catalog client instance for interacting with UC services.

Constructor Summary

Constructors

Constructor	Description
UCCommitCoordinatorClient(Map<String,String> conf, UCClient ucClient)

Method Summary

Modifier and Type	Method	Description
void	backfillToVersion(LogStore logStore, org.apache.hadoop.conf.Configuration hadoopConf, TableDescriptor tableDesc, long version, Long lastKnownBackfilledVersion)	API to ask the commit coordinator client to backfill all commits up to version and notify the commit coordinator.
CommitResponse	commit(LogStore logStore, org.apache.hadoop.conf.Configuration hadoopConf, TableDescriptor tableDesc, long commitVersion, Iterator<String> actions, UpdatedActions updatedActions)	API to commit the given set of actions to the table represented by logPath at the given commitVersion.
CommitResponse	commit(LogStore logStore, org.apache.hadoop.conf.Configuration hadoopConf, TableDescriptor tableDesc, long commitVersion, Iterator<String> actions, CatalogTrackedInfo catalogTrackedInfo, UpdatedActions updatedActions)
GetCommitsResponse	getCommits(TableDescriptor tableDesc, Long startVersion, Long endVersion)	API to get the unbackfilled commits for the table represented by the given logPath.
long	getLastKnownBackfilledVersion(long commitVersion, org.apache.hadoop.conf.Configuration hadoopConf, LogStore logStore, TableDescriptor tableDesc)	Find the last known backfilled version by doing a listing of the last BACKFILL_LISTING_OFFSET commits.
Map<String,String>	registerTable(org.apache.hadoop.fs.Path logPath, Optional<TableIdentifier> tableIdentifier, long currentVersion, AbstractMetadata currentMetadata, AbstractProtocol currentProtocol)	For UC, table registration is a no-op because we already contacted UC during table creation and that already obtained the necessary table config and added it to the metadata (this is for performance reasons and ease of use).
boolean	semanticEquals(CommitCoordinatorClient other)	Determines whether this CommitCoordinatorClient is semantically equal to another CommitCoordinatorClient.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

UC_TABLE_ID_KEY

public static final String UC_TABLE_ID_KEY

Key for identifying Unity Catalog table ID in `delta.coordinatedCommits.tableConf{-preview}`.

See Also:

• Constant Field Values

UC_TABLE_ID_KEY_OLD

public static final String UC_TABLE_ID_KEY_OLD

See Also:

• Constant Field Values

UC_METASTORE_ID_KEY

public static final String UC_METASTORE_ID_KEY

Key for identifying Unity Catalog metastore ID in `delta.coordinatedCommits.commitCoordinatorConf{-preview}`.

See Also:

• Constant Field Values

BACKFILL_LISTING_OFFSET

public static int BACKFILL_LISTING_OFFSET

Offset from current commit version for backfill listing optimization. Used to prevent expensive listings from version 0.

ucClient

public final UCClient ucClient

Unity Catalog client instance for interacting with UC services.

conf

public final Map<String,String> conf

Configuration map containing settings for the coordinator client.

Constructor Details

UCCommitCoordinatorClient

public UCCommitCoordinatorClient(Map<String,String> conf,
 UCClient ucClient)

Method Details

registerTable

public Map<String,String> registerTable(org.apache.hadoop.fs.Path logPath,
 Optional<TableIdentifier> tableIdentifier,
 long currentVersion,
 AbstractMetadata currentMetadata,
 AbstractProtocol currentProtocol)

For UC, table registration is a no-op because we already contacted UC during table creation and that already obtained the necessary table config and added it to the metadata (this is for performance reasons and ease of use). As a result, this method only verifies that the metadata has been added correct and is present. Otherwise, it throws an exception.

Specified by:

registerTable in interface CommitCoordinatorClient

Parameters:

logPath - The path to the delta log of the table that should be converted

tableIdentifier - The optional tableIdentifier for the table. Some commit coordinators may choose to make this compulsory and error out if this is not provided.

currentVersion - The currentTableVersion is the version of the table just before conversion. currentTableVersion + 1 represents the commit that will do the conversion. This must be backfilled atomically. currentTableVersion + 2 represents the first commit after conversion. This will go through the CommitCoordinatorClient and the client is free to choose when it wants to backfill this commit.

currentMetadata - The metadata of the table at currentTableVersion

currentProtocol - The protocol of the table at currentTableVersion

Returns:

A map of key-value pairs which is issued by the commit coordinator to identify the table. This should be stored in the table's metadata. This information needs to be passed to the CommitCoordinatorClient.commit(io.delta.storage.LogStore, org.apache.hadoop.conf.Configuration, io.delta.storage.commit.TableDescriptor, long, java.util.Iterator<java.lang.String>, io.delta.storage.commit.UpdatedActions), CommitCoordinatorClient.getCommits(io.delta.storage.commit.TableDescriptor, java.lang.Long, java.lang.Long), and CommitCoordinatorClient.backfillToVersion(io.delta.storage.LogStore, org.apache.hadoop.conf.Configuration, io.delta.storage.commit.TableDescriptor, long, java.lang.Long) APIs to identify the table.

getLastKnownBackfilledVersion

public long getLastKnownBackfilledVersion(long commitVersion,
 org.apache.hadoop.conf.Configuration hadoopConf,
 LogStore logStore,
 TableDescriptor tableDesc)

Find the last known backfilled version by doing a listing of the last BACKFILL_LISTING_OFFSET commits. If no backfilled commits are found among those, a UC call is made to get the oldest tracked commit in UC.

commit

public CommitResponse commit(LogStore logStore,
 org.apache.hadoop.conf.Configuration hadoopConf,
 TableDescriptor tableDesc,
 long commitVersion,
 Iterator<String> actions,
 UpdatedActions updatedActions)
                      throws CommitFailedException

Description copied from interface: CommitCoordinatorClient

API to commit the given set of actions to the table represented by logPath at the given commitVersion.

Specified by:

commit in interface CommitCoordinatorClient

Parameters:

logStore - The log store to use for writing the commit file.

hadoopConf - The Hadoop configuration required to access the file system.

tableDesc - The descriptor for the table.

commitVersion - The version of the commit that is being committed.

actions - The actions that need to be committed.

updatedActions - The commit info and any metadata or protocol changes that are made as part of this commit.

Returns:

CommitResponse which contains the file status of the committed commit file. If the commit is already backfilled, then the file status could be omitted from the response and the client could retrieve the information by itself.

Throws:

CommitFailedException - if the commit operation fails.

commit

public CommitResponse commit(LogStore logStore,
 org.apache.hadoop.conf.Configuration hadoopConf,
 TableDescriptor tableDesc,
 long commitVersion,
 Iterator<String> actions,
 CatalogTrackedInfo catalogTrackedInfo,
 UpdatedActions updatedActions)
                      throws CommitFailedException

Throws:

CommitFailedException

getCommits

public GetCommitsResponse getCommits(TableDescriptor tableDesc,
 Long startVersion,
 Long endVersion)

Description copied from interface: CommitCoordinatorClient

API to get the unbackfilled commits for the table represented by the given logPath. Commits older than startVersion or newer than endVersion (if given) are ignored. The returned commits are contiguous and in ascending version order. Note that the first version returned by this API may not be equal to startVersion. This happens when some versions starting from startVersion have already been backfilled and so the commit coordinator may have stopped tracking them. The returned latestTableVersion is the maximum commit version ratified by the commit coordinator. Note that returning latestTableVersion as -1 is acceptable only if the commit coordinator never ratified any version, i.e. it never accepted any unbackfilled commit.

Specified by:

getCommits in interface CommitCoordinatorClient

Parameters:

tableDesc - The descriptor for the table.

startVersion - The minimum version of the commit that should be returned. Can be null.

endVersion - The maximum version of the commit that should be returned. Can be null.

Returns:

GetCommitsResponse which has a list of Commits and the latestTableVersion which is tracked by CommitCoordinatorClient.

backfillToVersion

public void backfillToVersion(LogStore logStore,
 org.apache.hadoop.conf.Configuration hadoopConf,
 TableDescriptor tableDesc,
 long version,
 Long lastKnownBackfilledVersion)
                       throws IOException

Description copied from interface: CommitCoordinatorClient

API to ask the commit coordinator client to backfill all commits up to version and notify the commit coordinator. If this API returns successfully, that means the backfill must have been completed, although the commit coordinator may not be aware of it yet.

Specified by:

backfillToVersion in interface CommitCoordinatorClient

Parameters:

logStore - The log store to use for writing the backfilled commits.

hadoopConf - The Hadoop configuration required to access the file system.

tableDesc - The descriptor for the table.

version - The version till which the commit coordinator client should backfill.

lastKnownBackfilledVersion - The last known version that was backfilled before this API was called. If it is None or invalid, then the commit coordinator client should backfill from the beginning of the table. Can be null.

Throws:

IOException - if there is an IO error while backfilling the commits.

semanticEquals

public boolean semanticEquals(CommitCoordinatorClient other)

Description copied from interface: CommitCoordinatorClient

Determines whether this CommitCoordinatorClient is semantically equal to another CommitCoordinatorClient. Semantic equality is determined by each CommitCoordinatorClient implementation based on whether the two instances can be used interchangeably when invoking any of the CommitCoordinatorClient APIs, such as CommitCoordinatorClient.commit(io.delta.storage.LogStore, org.apache.hadoop.conf.Configuration, io.delta.storage.commit.TableDescriptor, long, java.util.Iterator<java.lang.String>, io.delta.storage.commit.UpdatedActions), CommitCoordinatorClient.getCommits(io.delta.storage.commit.TableDescriptor, java.lang.Long, java.lang.Long), etc. For example, both instances might be pointing to the same underlying endpoint.

Specified by:

semanticEquals in interface CommitCoordinatorClient