Package io.delta.kernel.internal

Class TransactionImpl

Object

io.delta.kernel.internal.TransactionImpl

All Implemented Interfaces:

Transaction

public class TransactionImpl
extends Object
implements Transaction

Field Summary

Fields

Modifier and Type	Field	Description
static final int	DEFAULT_READ_VERSION
static final int	DEFAULT_WRITE_VERSION

Constructor Summary

Constructors

Constructor	Description
TransactionImpl(boolean isCreateOrReplace, Path dataPath, Optional<SnapshotImpl> readSnapshotOpt, String engineInfo, Operation operation, Optional<Protocol> newProtocol, Optional<Metadata> newMetadata, Committer committer, Optional<SetTransaction> setTxnOpt, Optional<List<Column>> newClusteringColumnsOpt, Optional<Integer> maxRetriesOpt, int logCompactionInterval, Clock clock)

Method Summary

Modifier and Type	Method	Description
void	addDomainMetadata(String domain, String config)	Commit the provided domain metadata as part of this transaction.
void	addDomainMetadataInternal(String domain, String config)
TransactionCommitResult	commit(Engine engine, CloseableIterable<Row> dataActions)	Commit the transaction including the data action rows generated by Transaction.generateAppendActions(io.delta.kernel.engine.Engine, io.delta.kernel.data.Row, io.delta.kernel.utils.CloseableIterator<io.delta.kernel.utils.DataFileStatus>, io.delta.kernel.DataWriteContext).
Committer	getCommitter()
Path	getDataPath()
Optional<List<Column>>	getEffectiveClusteringColumns()
Path	getLogPath()
List<String>	getPartitionColumns(Engine engine)	Get the list of logical names of the partition columns.
Protocol	getProtocol()
long	getReadTableVersion()	Gets the latest version of the table used as the base of this transaction.
StructType	getSchema(Engine engine)	Get the schema of the table.
Optional<SetTransaction>	getSetTxnOpt()
static List<Column>	getStatisticsColumns(Row transactionState)	Get the part of the schema of the table that needs the statistics to be collected per file.
Row	getTransactionState(Engine engine)	Get the state of the transaction.
void	removeDomainMetadata(String domain)	Mark the domain metadata with identifier domain as removed in this transaction.
void	removeDomainMetadataInternal(String domain)
void	withCommitterProperties(Supplier<Map<String,String>> committerProperties)	Adds custom properties that will be passed through to the committer.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

DEFAULT_READ_VERSION

public static final int DEFAULT_READ_VERSION

See Also:

• Constant Field Values

DEFAULT_WRITE_VERSION

public static final int DEFAULT_WRITE_VERSION

See Also:

• Constant Field Values

Constructor Details

TransactionImpl

public TransactionImpl(boolean isCreateOrReplace,
 Path dataPath,
 Optional<SnapshotImpl> readSnapshotOpt,
 String engineInfo,
 Operation operation,
 Optional<Protocol> newProtocol,
 Optional<Metadata> newMetadata,
 Committer committer,
 Optional<SetTransaction> setTxnOpt,
 Optional<List<Column>> newClusteringColumnsOpt,
 Optional<Integer> maxRetriesOpt,
 int logCompactionInterval,
 Clock clock)

Method Details

getStatisticsColumns

public static List<Column> getStatisticsColumns(Row transactionState)

Get the part of the schema of the table that needs the statistics to be collected per file.

getTransactionState

public Row getTransactionState(Engine engine)

Description copied from interface: Transaction

Get the state of the transaction. The state helps Kernel do the transformations to logical data according to the Delta protocol and table features enabled on the table. The engine should use this at the data writer task to transform the logical data that the engine wants to write to the table in to physical data that goes in data files using Transaction.transformLogicalData(Engine, Row, CloseableIterator, Map)

Specified by:

getTransactionState in interface Transaction

getCommitter

public Committer getCommitter()

Specified by:

getCommitter in interface Transaction

Returns:

a committer that owns and controls commits to this table

getPartitionColumns

public List<String> getPartitionColumns(Engine engine)

Description copied from interface: Transaction

Get the list of logical names of the partition columns. This helps the connector to do physical partitioning of the data before asking the Kernel to stage the data per partition.

Specified by:

getPartitionColumns in interface Transaction

getSchema

public StructType getSchema(Engine engine)

Description copied from interface: Transaction

Get the schema of the table. If the connector is adding any data to the table through this transaction, it should have the same schema as the table schema.

Specified by:

getSchema in interface Transaction

getReadTableVersion

public long getReadTableVersion()

Description copied from interface: Transaction

Gets the latest version of the table used as the base of this transaction. This returns -1 when the table is being created in this transaction.

Specified by:

getReadTableVersion in interface Transaction

Returns:

The version of the table as of the beginning of this Transaction

withCommitterProperties

public void withCommitterProperties(Supplier<Map<String,String>> committerProperties)

Description copied from interface: Transaction

Adds custom properties that will be passed through to the committer. These properties allow connectors to inject catalog-specific metadata without Kernel inspection. Repeated calls to this method will overwrite any previously set properties.

Specified by:

withCommitterProperties in interface Transaction

addDomainMetadata

public void addDomainMetadata(String domain,
 String config)

Description copied from interface: Transaction

Commit the provided domain metadata as part of this transaction. If this is called more than once with the same domain the latest provided config will be committed in the transaction. Only user-controlled domains are allowed (aka. domains with a `delta.` prefix are not allowed). Adding and removing a domain with the same identifier in the same txn is not allowed. Adding domain metadata to a table that does not support the table feature is not allowed. To enable the table feature, make sure to call TransactionBuilder.withDomainMetadataSupported()

Specified by:

addDomainMetadata in interface Transaction

Parameters:

domain - the domain identifier

config - configuration string for this domain

removeDomainMetadata

public void removeDomainMetadata(String domain)

Description copied from interface: Transaction

Mark the domain metadata with identifier domain as removed in this transaction. If this domain does not exist in the latest version of the table, calling Transaction.commit(Engine, CloseableIterable) will throw a DomainDoesNotExistException. Adding and removing a domain with the same identifier in one txn is not allowed.

Specified by:

removeDomainMetadata in interface Transaction

Parameters:

domain - the domain identifier for the domain to remove

commit

public TransactionCommitResult commit(Engine engine,
 CloseableIterable<Row> dataActions)
                               throws ConcurrentWriteException

Description copied from interface: Transaction

Commit the transaction including the data action rows generated by Transaction.generateAppendActions(io.delta.kernel.engine.Engine, io.delta.kernel.data.Row, io.delta.kernel.utils.CloseableIterator<io.delta.kernel.utils.DataFileStatus>, io.delta.kernel.DataWriteContext).

Specified by:

commit in interface Transaction

Parameters:

engine - Engine instance.

dataActions - Iterable of data actions to commit. These data actions are generated by the Transaction.generateAppendActions(Engine, Row, CloseableIterator, DataWriteContext). The CloseableIterable allows the Kernel to access the list of actions multiple times (in case of retries to resolve the conflicts due to other writers to the table). Kernel provides a in-memory based implementation of CloseableIterable with utility API CloseableIterable.inMemoryIterable(CloseableIterator)

Returns:

TransactionCommitResult status of the successful transaction.

Throws:

ConcurrentWriteException - when the transaction has encountered a non-retryable conflicts or exceeded the maximum number of retries reached. The connector needs to rerun the query on top of the latest table state and retry the transaction.

addDomainMetadataInternal

@VisibleForTesting
public void addDomainMetadataInternal(String domain,
 String config)

removeDomainMetadataInternal

@VisibleForTesting
public void removeDomainMetadataInternal(String domain)

getDataPath

public Path getDataPath()

getLogPath

public Path getLogPath()

getProtocol

public Protocol getProtocol()

getSetTxnOpt

public Optional<SetTransaction> getSetTxnOpt()

getEffectiveClusteringColumns

public Optional<List<Column>> getEffectiveClusteringColumns()