org.apache.gobblin.util.filesystem

Interface DataFileVersionStrategy<T extends Comparable<T> & Serializable>

Type Parameters:

T - the type for the version objects. Must be comparable and serializable.

All Known Implementing Classes:

ModTimeDataFileVersionStrategy

public interface DataFileVersionStrategy<T extends Comparable<T> & Serializable>

An interface to set and get "versions" to data files. The version is a rough signature to the data contents. It allows data preserving functionality (like copy) to replicate the version independently of metadata with other semantics like file modification time. Examples where this might be useful is data syncing between two locations. Relying on modification times to detect data changes may lead to a feedback loop of copying: data gets created at location A at time 0, at time 1 data is copied to location B, sync mechanism might incorrectly believe that since mod time of location B is higher, it should be synced back to location A, etc. Required properties: - REPLICABLE: Two calls to `getVersion` on a file that has clearly not been modified must return the same version. - MONOTONOUS: The default version of a file is an increasing function of modification time. - CONSERVATIVE: If file f had its version last set to v, but the versioning implementation determines the file MIGHT have been modified and it chooses to return a version, it will return a value strictly larger than v. A common pattern to achieve monotonicity and conservativeness will be to invalidate the version of a data file if it is detected that the file was modified without updating the version (e.g. by a process which is unaware of versioning).

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	DataFileVersionStrategy.Characteristic Characteristics a DataFileVersionStrategy may have.
static interface	DataFileVersionStrategy.DataFileVersionFactory<T extends Comparable<T> & Serializable> A Factory for DataFileVersionStrategys.

Field Summary

Fields

Modifier and Type	Field and Description
static String	DATA_FILE_VERSION_STRATEGY_KEY
static String	DEFAULT_DATA_FILE_VERSION_STRATEGY

Method Summary

Modifier and Type	Method and Description
Set<DataFileVersionStrategy.Characteristic>	applicableCharacteristics()
T	getVersion(org.apache.hadoop.fs.Path path) Get the version of a path.
default boolean	hasCharacteristic(DataFileVersionStrategy.Characteristic characteristic)
static DataFileVersionStrategy	instantiateDataFileVersionStrategy(org.apache.hadoop.fs.FileSystem fs, com.typesafe.config.Config config) Instantiate a DataFileVersionStrategy according to input configuration.
boolean	setDefaultVersion(org.apache.hadoop.fs.Path path) Set the version of a path to a value automatically set by the versioning system.
boolean	setVersion(org.apache.hadoop.fs.Path path, T version) Set the version of a path to a specific version (generally replicated from another path).

Field Detail

DATA_FILE_VERSION_STRATEGY_KEY

static final String DATA_FILE_VERSION_STRATEGY_KEY

See Also:

Constant Field Values

DEFAULT_DATA_FILE_VERSION_STRATEGY

static final String DEFAULT_DATA_FILE_VERSION_STRATEGY

See Also:

Constant Field Values

Method Detail

instantiateDataFileVersionStrategy

static DataFileVersionStrategy instantiateDataFileVersionStrategy(org.apache.hadoop.fs.FileSystem fs,
                                                                  com.typesafe.config.Config config)
                                                           throws IOException

Instantiate a DataFileVersionStrategy according to input configuration.

Throws:

IOException

getVersion

T getVersion(org.apache.hadoop.fs.Path path)
      throws IOException

Get the version of a path.

Throws:

IOException

setVersion

boolean setVersion(org.apache.hadoop.fs.Path path,
                   T version)
            throws IOException

Set the version of a path to a specific version (generally replicated from another path).

Returns:

false if the version is not settable.

Throws:

IOException - if the version is settable but could not be set successfully.

setDefaultVersion

boolean setDefaultVersion(org.apache.hadoop.fs.Path path)
                   throws IOException

Set the version of a path to a value automatically set by the versioning system. Note this call must respect the monotonicity requirement.

Returns:

false if the version is not settable.

Throws:

IOException - if the version is settable but could not be set successfully.

applicableCharacteristics

Set<DataFileVersionStrategy.Characteristic> applicableCharacteristics()

Returns:

The list of optional characteristics this DataFileVersionStrategy satisfies.

hasCharacteristic

default boolean hasCharacteristic(DataFileVersionStrategy.Characteristic characteristic)

Returns:

whether this implementation have the specified characteristic.