org.apache.hudi.metadata

Class FileSystemBackedTableMetadata

java.lang.Object

org.apache.hudi.metadata.AbstractHoodieTableMetadata

org.apache.hudi.metadata.FileSystemBackedTableMetadata

All Implemented Interfaces:

Serializable, AutoCloseable, HoodieTableMetadata

public class FileSystemBackedTableMetadata
extends AbstractHoodieTableMetadata

Implementation of HoodieTableMetadata based file-system-backed table metadata.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.hudi.metadata.AbstractHoodieTableMetadata

caseSensitive, dataBasePath, engineContext, storage, storageConf

Fields inherited from interface org.apache.hudi.metadata.HoodieTableMetadata

EMPTY_PARTITION_NAME, METADATA_TABLE_NAME_SUFFIX, NON_PARTITIONED_NAME, RECORDKEY_PARTITION_LIST, SOLO_COMMIT_TIMESTAMP

Constructor Summary

Constructors

Constructor and Description
FileSystemBackedTableMetadata(HoodieEngineContext engineContext, HoodieStorage storage, String datasetBasePath, boolean assumeDatePartitioning)
FileSystemBackedTableMetadata(HoodieEngineContext engineContext, HoodieTableConfig tableConfig, HoodieStorage storage, String datasetBasePath, boolean assumeDatePartitioning)

Method Summary

Modifier and Type	Method and Description
void	close()
List<StoragePathInfo>	getAllFilesInPartition(StoragePath partitionPath) Fetch all the files at the given partition path, per the latest snapshot of the metadata.
Map<String,List<StoragePathInfo>>	getAllFilesInPartitions(Collection<String> partitionPaths) Fetch all files for given partition paths.
List<String>	getAllPartitionPaths() Fetch list of all partition paths, per the latest snapshot of the metadata.
Option<BloomFilter>	getBloomFilter(String partitionName, String fileName) Get the bloom filter for the FileID from the metadata table.
Map<Pair<String,String>,BloomFilter>	getBloomFilters(List<Pair<String,String>> partitionNameFileNameList) Get bloom filters for files from the metadata table index.
Map<Pair<String,String>,HoodieMetadataColumnStats>	getColumnStats(List<Pair<String,String>> partitionNameFileNameList, String columnName) Get column stats for files from the metadata table index.
Option<String>	getLatestCompactionTime() Returns the timestamp of the latest compaction.
int	getNumFileGroupsForPartition(MetadataPartitionType partition) Returns the number of shards in a metadata table partition.
List<String>	getPartitionPathWithPathPrefixes(List<String> relativePathPrefixes) Fetches all partition paths that are the sub-directories of the list of provided (relative) paths.
List<String>	getPartitionPathWithPathPrefixUsingFilterExpression(List<String> relativePathPrefixes, Types.RecordType partitionFields, Expression expression) Retrieve the paths of partitions under the provided sub-directories, and try to filter these partitions using the provided Expression.
HoodieData<HoodieRecord<HoodieMetadataPayload>>	getRecordsByKeyPrefixes(List<String> keyPrefixes, String partitionName, boolean shouldLoadInMemory) Fetch records by key prefixes.
HoodieStorage	getStorage()
Option<String>	getSyncedInstantTime() Get the instant time to which the metadata is synced w.r.t data timeline.
Map<String,HoodieRecordGlobalLocation>	readRecordIndex(List<String> recordKeys) Returns the location of record keys which are found in the record index.
void	reset() Clear the states of the table metadata.

Methods inherited from class org.apache.hudi.metadata.AbstractHoodieTableMetadata

extractPartitionValues, getPathPartitionLevel

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hudi.metadata.HoodieTableMetadata

create, create, createFSBackedTableMetadata, createHoodieBackedTableMetadata, getDatasetBasePath, getDataTableBasePathFromMetadataTable, getMetadataTableBasePath, getMetadataTableBasePath, isMetadataTable

Constructor Detail

FileSystemBackedTableMetadata

public FileSystemBackedTableMetadata(HoodieEngineContext engineContext,
                                     HoodieTableConfig tableConfig,
                                     HoodieStorage storage,
                                     String datasetBasePath,
                                     boolean assumeDatePartitioning)

FileSystemBackedTableMetadata

public FileSystemBackedTableMetadata(HoodieEngineContext engineContext,
                                     HoodieStorage storage,
                                     String datasetBasePath,
                                     boolean assumeDatePartitioning)

Method Detail

getStorage

public HoodieStorage getStorage()

getAllFilesInPartition

public List<StoragePathInfo> getAllFilesInPartition(StoragePath partitionPath)
                                             throws IOException

Description copied from interface: HoodieTableMetadata

Fetch all the files at the given partition path, per the latest snapshot of the metadata.

Throws:

IOException

getAllPartitionPaths

public List<String> getAllPartitionPaths()
                                  throws IOException

Description copied from interface: HoodieTableMetadata

Fetch list of all partition paths, per the latest snapshot of the metadata.

Throws:

IOException

getPartitionPathWithPathPrefixUsingFilterExpression

public List<String> getPartitionPathWithPathPrefixUsingFilterExpression(List<String> relativePathPrefixes,
                                                                        Types.RecordType partitionFields,
                                                                        Expression expression)
                                                                 throws IOException

Description copied from interface: HoodieTableMetadata

Retrieve the paths of partitions under the provided sub-directories, and try to filter these partitions using the provided Expression.

Throws:

IOException

getPartitionPathWithPathPrefixes

public List<String> getPartitionPathWithPathPrefixes(List<String> relativePathPrefixes)

Description copied from interface: HoodieTableMetadata

Fetches all partition paths that are the sub-directories of the list of provided (relative) paths.

E.g., Table has partition 4 partitions: year=2022/month=08/day=30, year=2022/month=08/day=31, year=2022/month=07/day=03, year=2022/month=07/day=04 The relative path "year=2022" returns all partitions, while the relative path "year=2022/month=07" returns only two partitions.

getAllFilesInPartitions

public Map<String,List<StoragePathInfo>> getAllFilesInPartitions(Collection<String> partitionPaths)
                                                          throws IOException

Description copied from interface: HoodieTableMetadata

Fetch all files for given partition paths. NOTE: Absolute partition paths are expected here

Throws:

IOException

getSyncedInstantTime

public Option<String> getSyncedInstantTime()

Description copied from interface: HoodieTableMetadata

Get the instant time to which the metadata is synced w.r.t data timeline.

getLatestCompactionTime

public Option<String> getLatestCompactionTime()

Description copied from interface: HoodieTableMetadata

Returns the timestamp of the latest compaction.

close

public void close()
           throws Exception

Throws:

Exception

reset

public void reset()

Description copied from interface: HoodieTableMetadata

Clear the states of the table metadata.

getBloomFilter

public Option<BloomFilter> getBloomFilter(String partitionName,
                                          String fileName)
                                   throws HoodieMetadataException

Description copied from interface: HoodieTableMetadata

Get the bloom filter for the FileID from the metadata table.

Parameters:

partitionName - - Partition name

fileName - - File name for which bloom filter needs to be retrieved

Returns:

BloomFilter if available, otherwise empty

Throws:

HoodieMetadataException

getBloomFilters

public Map<Pair<String,String>,BloomFilter> getBloomFilters(List<Pair<String,String>> partitionNameFileNameList)
                                                     throws HoodieMetadataException

Description copied from interface: HoodieTableMetadata

Get bloom filters for files from the metadata table index.

Parameters:

partitionNameFileNameList - - List of partition and file name pair for which bloom filters need to be retrieved

Returns:

Map of partition file name pair to its bloom filter

Throws:

HoodieMetadataException

getColumnStats

public Map<Pair<String,String>,HoodieMetadataColumnStats> getColumnStats(List<Pair<String,String>> partitionNameFileNameList,
                                                                         String columnName)
                                                                  throws HoodieMetadataException

Description copied from interface: HoodieTableMetadata

Get column stats for files from the metadata table index.

Parameters:

partitionNameFileNameList - - List of partition and file name pair for which bloom filters need to be retrieved

columnName - - Column name for which stats are needed

Returns:

Map of partition and file name pair to its column stats

Throws:

HoodieMetadataException

getRecordsByKeyPrefixes

public HoodieData<HoodieRecord<HoodieMetadataPayload>> getRecordsByKeyPrefixes(List<String> keyPrefixes,
                                                                               String partitionName,
                                                                               boolean shouldLoadInMemory)

Description copied from interface: HoodieTableMetadata

Fetch records by key prefixes. Key prefix passed is expected to match the same prefix as stored in Metadata table partitions. For eg, in case of col stats partition, actual keys in metadata partition is encoded values of column name, partition name and file name. So, key prefixes passed to this method is expected to be encoded already.

Parameters:

keyPrefixes - list of key prefixes for which interested records are looked up for.

partitionName - partition name in metadata table where the records are looked up for.

Returns:

HoodieData of HoodieRecords with records matching the passed in key prefixes.

readRecordIndex

public Map<String,HoodieRecordGlobalLocation> readRecordIndex(List<String> recordKeys)

Description copied from interface: HoodieTableMetadata

Returns the location of record keys which are found in the record index. Records that are not found are ignored and wont be part of map object that is returned.

getNumFileGroupsForPartition

public int getNumFileGroupsForPartition(MetadataPartitionType partition)

Description copied from interface: HoodieTableMetadata

Returns the number of shards in a metadata table partition.