org.apache.hudi.common.table.view

Class PriorityBasedFileSystemView

java.lang.Object

org.apache.hudi.common.table.view.PriorityBasedFileSystemView

All Implemented Interfaces:

Serializable, SyncableFileSystemView, TableFileSystemView, TableFileSystemView.BaseFileOnlyView, TableFileSystemView.BaseFileOnlyViewWithLatestSlice, TableFileSystemView.SliceView, TableFileSystemView.SliceViewWithLatestSlice

public class PriorityBasedFileSystemView
extends Object
implements SyncableFileSystemView, Serializable

A file system view which proxies request to a preferred File System View implementation. In case of error, flip all subsequent calls to a backup file-system view implementation.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hudi.common.table.view.TableFileSystemView

TableFileSystemView.BaseFileOnlyView, TableFileSystemView.BaseFileOnlyViewWithLatestSlice, TableFileSystemView.SliceView, TableFileSystemView.SliceViewWithLatestSlice

Constructor Summary

Constructors

Constructor and Description
PriorityBasedFileSystemView(SyncableFileSystemView preferredView, SyncableFileSystemView secondaryView)

Method Summary

Modifier and Type	Method and Description
void	close() Allow View to release resources and close.
Stream<HoodieBaseFile>	getAllBaseFiles(String partitionPath) Stream all the data file versions grouped by FileId for a given partition.
Stream<HoodieFileGroup>	getAllFileGroups(String partitionPath) Stream all the file groups for a given partition.
Stream<HoodieFileGroup>	getAllFileGroupsStateless(String partitionPath) Stream all the file groups for a given partition without caching the file group mappings.
Stream<FileSlice>	getAllFileSlices(String partitionPath) Stream all the file slices for a given partition, latest or not.
Map<String,Stream<HoodieBaseFile>>	getAllLatestBaseFilesBeforeOrOn(String maxCommitTime) Streams the latest version base files in all partitions with precondition that commitTime(file) before maxCommitTime.
Map<String,Stream<FileSlice>>	getAllLatestFileSlicesBeforeOrOn(String maxCommitTime) Stream all latest file slices with precondition that commitTime(file) before maxCommitTime.
Stream<HoodieFileGroup>	getAllReplacedFileGroups(String partitionPath) Stream all the replaced file groups for given partition.
Option<HoodieBaseFile>	getBaseFileOn(String partitionPath, String instantTime, String fileId) Get the version of data file matching the instant time in the given partition.
Stream<Pair<HoodieFileGroupId,HoodieInstant>>	getFileGroupsInPendingClustering() Filegroups that are in pending clustering.
Option<HoodieInstant>	getLastInstant() Last Known Instant on which the view is built.
Option<HoodieBaseFile>	getLatestBaseFile(String partitionPath, String fileId) Get Latest data file for a partition and file-Id.
Stream<HoodieBaseFile>	getLatestBaseFiles() Stream all the latest data files, in the file system view.
Stream<HoodieBaseFile>	getLatestBaseFiles(String partitionPath) Stream all the latest data files in the given partition.
Stream<HoodieBaseFile>	getLatestBaseFilesBeforeOrOn(String partitionPath, String maxCommitTime) Stream all the latest version data files in the given partition with precondition that commitTime(file) before maxCommitTime.
Stream<HoodieBaseFile>	getLatestBaseFilesInRange(List<String> commitsToReturn) Stream all the latest data files pass.
Option<FileSlice>	getLatestFileSlice(String partitionPath, String fileId) Get Latest File Slice for a given fileId in a given partition.
Stream<FileSlice>	getLatestFileSliceInRange(List<String> commitsToReturn) Stream all the latest file slices, in the given range.
Stream<FileSlice>	getLatestFileSlices(String partitionPath) Stream all the latest file slices in the given partition.
Stream<FileSlice>	getLatestFileSlicesBeforeOrOn(String partitionPath, String maxCommitTime, boolean includeFileSlicesInPendingCompaction) Stream all latest file slices in given partition with precondition that commitTime(file) before maxCommitTime.
Stream<FileSlice>	getLatestFileSlicesStateless(String partitionPath) Stream all the latest file slices in the given partition without caching the file group mappings.
Stream<FileSlice>	getLatestMergedFileSlicesBeforeOrOn(String partitionPath, String maxInstantTime) Stream all "merged" file-slices before on an instant time If a file-group has a pending compaction request, the file-slice before and after compaction request instant is merged and returned.
Stream<FileSlice>	getLatestUnCompactedFileSlices(String partitionPath) Stream all the latest uncompacted file slices in the given partition.
Stream<Pair<String,CompactionOperation>>	getPendingCompactionOperations() Return Pending Compaction Operations.
Stream<Pair<String,CompactionOperation>>	getPendingLogCompactionOperations() Return Pending Compaction Operations.
SyncableFileSystemView	getPreferredView()
Stream<HoodieFileGroup>	getReplacedFileGroupsAfterOrOn(String minCommitTime, String partitionPath) Stream all the replaced file groups after or on minCommitTime.
Stream<HoodieFileGroup>	getReplacedFileGroupsBefore(String maxCommitTime, String partitionPath) Stream all the replaced file groups before maxCommitTime for given partition.
Stream<HoodieFileGroup>	getReplacedFileGroupsBeforeOrOn(String maxCommitTime, String partitionPath) Stream all the replaced file groups before or on maxCommitTime for given partition.
SyncableFileSystemView	getSecondaryView()
HoodieTimeline	getTimeline() Timeline corresponding to the view.
void	loadAllPartitions() Load all partition and file slices into view
void	loadPartitions(List<String> partitionPaths) Load all partition and file slices into view for the provided partition paths
void	reset() Reset View so that they can be refreshed.
void	sync() Read the latest timeline and refresh the file-system view to match the current state of the file-system.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PriorityBasedFileSystemView

public PriorityBasedFileSystemView(SyncableFileSystemView preferredView,
                                   SyncableFileSystemView secondaryView)

Method Detail

getLatestBaseFiles

public Stream<HoodieBaseFile> getLatestBaseFiles(String partitionPath)

Description copied from interface: TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Stream all the latest data files in the given partition.

Specified by:

getLatestBaseFiles in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice

getLatestBaseFiles

public Stream<HoodieBaseFile> getLatestBaseFiles()

Description copied from interface: TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Stream all the latest data files, in the file system view.

Specified by:

getLatestBaseFiles in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice

getLatestBaseFilesBeforeOrOn

public Stream<HoodieBaseFile> getLatestBaseFilesBeforeOrOn(String partitionPath,
                                                           String maxCommitTime)

Description copied from interface: TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Stream all the latest version data files in the given partition with precondition that commitTime(file) before maxCommitTime.

Specified by:

getLatestBaseFilesBeforeOrOn in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice

getAllLatestBaseFilesBeforeOrOn

public Map<String,Stream<HoodieBaseFile>> getAllLatestBaseFilesBeforeOrOn(String maxCommitTime)

Description copied from interface: TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Streams the latest version base files in all partitions with precondition that commitTime(file) before maxCommitTime.

Specified by:

getAllLatestBaseFilesBeforeOrOn in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Parameters:

maxCommitTime - The max commit time to consider.

Returns:

A Map of partition path to the latest version base files before or on the commit time

getLatestBaseFile

public Option<HoodieBaseFile> getLatestBaseFile(String partitionPath,
                                                String fileId)

Description copied from interface: TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Get Latest data file for a partition and file-Id.

Specified by:

getLatestBaseFile in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice

getBaseFileOn

public Option<HoodieBaseFile> getBaseFileOn(String partitionPath,
                                            String instantTime,
                                            String fileId)

Description copied from interface: TableFileSystemView.BaseFileOnlyView

Get the version of data file matching the instant time in the given partition.

Specified by:

getBaseFileOn in interface TableFileSystemView.BaseFileOnlyView

getLatestBaseFilesInRange

public Stream<HoodieBaseFile> getLatestBaseFilesInRange(List<String> commitsToReturn)

Description copied from interface: TableFileSystemView.BaseFileOnlyViewWithLatestSlice

Stream all the latest data files pass.

Specified by:

getLatestBaseFilesInRange in interface TableFileSystemView.BaseFileOnlyViewWithLatestSlice

loadAllPartitions

public void loadAllPartitions()

Description copied from interface: TableFileSystemView

Load all partition and file slices into view

Specified by:

loadAllPartitions in interface TableFileSystemView

loadPartitions

public void loadPartitions(List<String> partitionPaths)

Description copied from interface: TableFileSystemView

Load all partition and file slices into view for the provided partition paths

Specified by:

loadPartitions in interface TableFileSystemView

Parameters:

partitionPaths - List of partition paths to load

getAllBaseFiles

public Stream<HoodieBaseFile> getAllBaseFiles(String partitionPath)

Description copied from interface: TableFileSystemView.BaseFileOnlyView

Stream all the data file versions grouped by FileId for a given partition.

Specified by:

getAllBaseFiles in interface TableFileSystemView.BaseFileOnlyView

getLatestFileSlices

public Stream<FileSlice> getLatestFileSlices(String partitionPath)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all the latest file slices in the given partition.

Specified by:

getLatestFileSlices in interface TableFileSystemView.SliceViewWithLatestSlice

getLatestFileSlicesStateless

public Stream<FileSlice> getLatestFileSlicesStateless(String partitionPath)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all the latest file slices in the given partition without caching the file group mappings.

This is useful for some table services such as compaction and clustering, these services may search around the files to clean within some ancient data partitions, if there triggers a full table service for enormous number of partitions, the cache could cause a huge memory pressure to the timeline server which induces an OOM exception.

The caching of these file groups does not benefit to writers most often because the writers write to recent data partitions usually.

Specified by:

getLatestFileSlicesStateless in interface TableFileSystemView.SliceViewWithLatestSlice

getLatestUnCompactedFileSlices

public Stream<FileSlice> getLatestUnCompactedFileSlices(String partitionPath)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all the latest uncompacted file slices in the given partition.

Specified by:

getLatestUnCompactedFileSlices in interface TableFileSystemView.SliceViewWithLatestSlice

getLatestFileSlicesBeforeOrOn

public Stream<FileSlice> getLatestFileSlicesBeforeOrOn(String partitionPath,
                                                       String maxCommitTime,
                                                       boolean includeFileSlicesInPendingCompaction)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all latest file slices in given partition with precondition that commitTime(file) before maxCommitTime.

Specified by:

getLatestFileSlicesBeforeOrOn in interface TableFileSystemView.SliceViewWithLatestSlice

Parameters:

partitionPath - Partition path

maxCommitTime - Max Instant Time

includeFileSlicesInPendingCompaction - include file-slices that are in pending compaction

getAllLatestFileSlicesBeforeOrOn

public Map<String,Stream<FileSlice>> getAllLatestFileSlicesBeforeOrOn(String maxCommitTime)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all latest file slices with precondition that commitTime(file) before maxCommitTime.

Specified by:

getAllLatestFileSlicesBeforeOrOn in interface TableFileSystemView.SliceViewWithLatestSlice

Parameters:

maxCommitTime - Max Instant Time

Returns:

A Map of partition path to the latest file slices before maxCommitTime.

getLatestMergedFileSlicesBeforeOrOn

public Stream<FileSlice> getLatestMergedFileSlicesBeforeOrOn(String partitionPath,
                                                             String maxInstantTime)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all "merged" file-slices before on an instant time If a file-group has a pending compaction request, the file-slice before and after compaction request instant is merged and returned.

Specified by:

getLatestMergedFileSlicesBeforeOrOn in interface TableFileSystemView.SliceViewWithLatestSlice

Parameters:

partitionPath - Partition Path

maxInstantTime - Max Instant Time

Returns:

getLatestFileSliceInRange

public Stream<FileSlice> getLatestFileSliceInRange(List<String> commitsToReturn)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Stream all the latest file slices, in the given range.

Specified by:

getLatestFileSliceInRange in interface TableFileSystemView.SliceViewWithLatestSlice

getAllFileSlices

public Stream<FileSlice> getAllFileSlices(String partitionPath)

Description copied from interface: TableFileSystemView.SliceView

Stream all the file slices for a given partition, latest or not.

Specified by:

getAllFileSlices in interface TableFileSystemView.SliceView

getAllFileGroups

public Stream<HoodieFileGroup> getAllFileGroups(String partitionPath)

Description copied from interface: TableFileSystemView

Stream all the file groups for a given partition.

Specified by:

getAllFileGroups in interface TableFileSystemView

getAllFileGroupsStateless

public Stream<HoodieFileGroup> getAllFileGroupsStateless(String partitionPath)

Description copied from interface: TableFileSystemView

Stream all the file groups for a given partition without caching the file group mappings.

This is useful for some table services such as cleaning, the cleaning service may search around the files to clean within some ancient data partitions, if there triggers a full table cleaning for enormous number of partitions, the cache could cause a huge memory pressure to the timeline server which induces an OOM exception.

The caching of these file groups does not benefit to writers most often because the writers write to recent data partitions usually.

Specified by:

getAllFileGroupsStateless in interface TableFileSystemView

getReplacedFileGroupsBeforeOrOn

public Stream<HoodieFileGroup> getReplacedFileGroupsBeforeOrOn(String maxCommitTime,
                                                               String partitionPath)

Description copied from interface: TableFileSystemView

Stream all the replaced file groups before or on maxCommitTime for given partition.

Specified by:

getReplacedFileGroupsBeforeOrOn in interface TableFileSystemView

getReplacedFileGroupsBefore

public Stream<HoodieFileGroup> getReplacedFileGroupsBefore(String maxCommitTime,
                                                           String partitionPath)

Description copied from interface: TableFileSystemView

Stream all the replaced file groups before maxCommitTime for given partition.

Specified by:

getReplacedFileGroupsBefore in interface TableFileSystemView

getReplacedFileGroupsAfterOrOn

public Stream<HoodieFileGroup> getReplacedFileGroupsAfterOrOn(String minCommitTime,
                                                              String partitionPath)

Description copied from interface: TableFileSystemView

Stream all the replaced file groups after or on minCommitTime.

Specified by:

getReplacedFileGroupsAfterOrOn in interface TableFileSystemView

getAllReplacedFileGroups

public Stream<HoodieFileGroup> getAllReplacedFileGroups(String partitionPath)

Description copied from interface: TableFileSystemView

Stream all the replaced file groups for given partition.

Specified by:

getAllReplacedFileGroups in interface TableFileSystemView

getPendingCompactionOperations

public Stream<Pair<String,CompactionOperation>> getPendingCompactionOperations()

Description copied from interface: TableFileSystemView

Return Pending Compaction Operations.

Specified by:

getPendingCompactionOperations in interface TableFileSystemView

Returns:

Stream>

getPendingLogCompactionOperations

public Stream<Pair<String,CompactionOperation>> getPendingLogCompactionOperations()

Description copied from interface: TableFileSystemView

Return Pending Compaction Operations.

Specified by:

getPendingLogCompactionOperations in interface TableFileSystemView

Returns:

Stream>

getFileGroupsInPendingClustering

public Stream<Pair<HoodieFileGroupId,HoodieInstant>> getFileGroupsInPendingClustering()

Description copied from interface: TableFileSystemView

Filegroups that are in pending clustering.

Specified by:

getFileGroupsInPendingClustering in interface TableFileSystemView

close

public void close()

Description copied from interface: SyncableFileSystemView

Allow View to release resources and close.

Specified by:

close in interface SyncableFileSystemView

reset

public void reset()

Description copied from interface: SyncableFileSystemView

Reset View so that they can be refreshed.

Specified by:

reset in interface SyncableFileSystemView

getLastInstant

public Option<HoodieInstant> getLastInstant()

Description copied from interface: TableFileSystemView

Last Known Instant on which the view is built.

Specified by:

getLastInstant in interface TableFileSystemView

getTimeline

public HoodieTimeline getTimeline()

Description copied from interface: TableFileSystemView

Timeline corresponding to the view.

Specified by:

getTimeline in interface TableFileSystemView

sync

public void sync()

Description copied from interface: SyncableFileSystemView

Read the latest timeline and refresh the file-system view to match the current state of the file-system. The refresh can either be done incrementally (from reading file-slices in metadata files) or from scratch by resetting view storage.

Specified by:

sync in interface SyncableFileSystemView

getLatestFileSlice

public Option<FileSlice> getLatestFileSlice(String partitionPath,
                                            String fileId)

Description copied from interface: TableFileSystemView.SliceViewWithLatestSlice

Get Latest File Slice for a given fileId in a given partition.

Specified by:

getLatestFileSlice in interface TableFileSystemView.SliceViewWithLatestSlice

getPreferredView

public SyncableFileSystemView getPreferredView()

getSecondaryView

public SyncableFileSystemView getSecondaryView()