org.apache.hudi.common.util

Class MarkerUtils

java.lang.Object

org.apache.hudi.common.util.MarkerUtils

public class MarkerUtils
extends Object

A utility class for marker related operations.

Field Summary

Fields

Modifier and Type	Field and Description
static String	MARKER_TYPE_FILENAME
static String	MARKERS_FILENAME_PREFIX

Constructor Summary

Constructors

Constructor and Description
MarkerUtils()

Method Summary

Modifier and Type	Method and Description
static void	deleteMarkerTypeFile(HoodieStorage storage, String markerDir) Deletes `MARKERS.type` file.
static boolean	doesMarkerTypeFileExist(HoodieStorage storage, String markerDir)
static List<StoragePath>	getAllMarkerDir(StoragePath tempPath, HoodieStorage storage) Gets all marker directories.
static List<String>	getCandidateInstants(HoodieActiveTimeline activeTimeline, List<StoragePath> instants, String currentInstantTime, long maxAllowableHeartbeatIntervalInMs, HoodieStorage storage, String basePath) Get Candidate Instant to do conflict checking: 1.
static boolean	hasCommitConflict(HoodieActiveTimeline activeTimeline, Set<String> currentFileIDs, Set<HoodieInstant> completedCommitInstants) Whether there is write conflict with completed commit among multiple writers.
static String	makerToPartitionAndFileID(String marker) Get fileID from full marker path, for example: 20210623/0/20210825/932a86d9-5c1d-44c7-ac99-cb88b8ef8478-0_85-15-1390_20220620181735781.parquet.marker.MERGE ==> get 20210623/0/20210825/932a86d9-5c1d-44c7-ac99-cb88b8ef8478-0
static String	markerDirToInstantTime(String marker) Get instantTime from full marker path, for example: /var/folders/t3/th1dw75d0yz2x2k2qt6ys9zh0000gp/T/junit6502909693741900820/dataset/.hoodie/.temp/003 ==> 003
static Set<String>	readMarkersFromFile(StoragePath markersFilePath, StorageConfiguration<?> conf) Reads the markers stored in the underlying file.
static Set<String>	readMarkersFromFile(StoragePath markersFilePath, StorageConfiguration<?> conf, boolean ignoreException) Reads the markers stored in the underlying file.
static Option<MarkerType>	readMarkerType(HoodieStorage storage, String markerDir) Reads the marker type from `MARKERS.type` file.
static Map<String,Set<String>>	readTimelineServerBasedMarkersFromFileSystem(String markerDir, HoodieStorage storage, HoodieEngineContext context, int parallelism) Reads files containing the markers written by timeline-server-based marker mechanism.
static String	stripMarkerFolderPrefix(String fullMarkerPath, String markerDir) Strips the marker folder prefix of any file path under the marker directory.
static String	stripMarkerFolderPrefix(String fullMarkerPath, String basePath, String instantTime) Strips the folder prefix of the marker file path corresponding to a data file.
static void	writeMarkerTypeToFile(MarkerType markerType, HoodieStorage storage, String markerDir) Writes the marker type to the file `MARKERS.type`.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MARKERS_FILENAME_PREFIX

public static final String MARKERS_FILENAME_PREFIX

See Also:

Constant Field Values

MARKER_TYPE_FILENAME

public static final String MARKER_TYPE_FILENAME

See Also:

Constant Field Values

Constructor Detail

MarkerUtils

public MarkerUtils()

Method Detail

stripMarkerFolderPrefix

public static String stripMarkerFolderPrefix(String fullMarkerPath,
                                             String basePath,
                                             String instantTime)

Strips the folder prefix of the marker file path corresponding to a data file.

Parameters:

fullMarkerPath - the full path of the marker file

basePath - the base path

instantTime - instant of interest

Returns:

marker file name

stripMarkerFolderPrefix

public static String stripMarkerFolderPrefix(String fullMarkerPath,
                                             String markerDir)

Strips the marker folder prefix of any file path under the marker directory.

Parameters:

fullMarkerPath - the full path of the file

markerDir - marker directory

Returns:

file name

doesMarkerTypeFileExist

public static boolean doesMarkerTypeFileExist(HoodieStorage storage,
                                              String markerDir)
                                       throws IOException

Parameters:

storage - HoodieStorage to use.

markerDir - marker directory.

Returns:

true if the MARKERS.type file exists; false otherwise.

Throws:

IOException

readMarkerType

public static Option<MarkerType> readMarkerType(HoodieStorage storage,
                                                String markerDir)

Reads the marker type from `MARKERS.type` file.

Parameters:

storage - HoodieStorage to use.

markerDir - marker directory.

Returns:

the marker type, or empty if the marker type file does not exist.

writeMarkerTypeToFile

public static void writeMarkerTypeToFile(MarkerType markerType,
                                         HoodieStorage storage,
                                         String markerDir)

Writes the marker type to the file `MARKERS.type`.

Parameters:

markerType - marker type.

storage - HoodieStorage to use.

markerDir - marker directory.

deleteMarkerTypeFile

public static void deleteMarkerTypeFile(HoodieStorage storage,
                                        String markerDir)

Deletes `MARKERS.type` file.

Parameters:

storage - HoodieStorage to use.

markerDir - marker directory.

readTimelineServerBasedMarkersFromFileSystem

public static Map<String,Set<String>> readTimelineServerBasedMarkersFromFileSystem(String markerDir,
                                                                                   HoodieStorage storage,
                                                                                   HoodieEngineContext context,
                                                                                   int parallelism)

Reads files containing the markers written by timeline-server-based marker mechanism.

Parameters:

markerDir - marker directory.

storage - file system to use.

context - instance of HoodieEngineContext to use

parallelism - parallelism to use

Returns:

A Map of file name to the set of markers stored in the file.

readMarkersFromFile

public static Set<String> readMarkersFromFile(StoragePath markersFilePath,
                                              StorageConfiguration<?> conf)

Reads the markers stored in the underlying file.

Parameters:

markersFilePath - file path for the markers

conf - storage config

Returns:

markers in a Set of String.

readMarkersFromFile

public static Set<String> readMarkersFromFile(StoragePath markersFilePath,
                                              StorageConfiguration<?> conf,
                                              boolean ignoreException)

Reads the markers stored in the underlying file.

Parameters:

markersFilePath - File path for the markers.

conf - storage config.

ignoreException - Whether to ignore IOException.

Returns:

Markers in a Set of String.

getAllMarkerDir

public static List<StoragePath> getAllMarkerDir(StoragePath tempPath,
                                                HoodieStorage storage)
                                         throws IOException

Gets all marker directories.

Parameters:

tempPath - Temporary folder under .hoodie.

storage - File system to use.

Returns:

All marker directories.

Throws:

IOException - upon error.

hasCommitConflict

public static boolean hasCommitConflict(HoodieActiveTimeline activeTimeline,
                                        Set<String> currentFileIDs,
                                        Set<HoodieInstant> completedCommitInstants)

Whether there is write conflict with completed commit among multiple writers.

Parameters:

activeTimeline - Active timeline.

currentFileIDs - Current set of file IDs.

completedCommitInstants - Completed commits.

Returns:

true if the conflict is detected; false otherwise.

getCandidateInstants

public static List<String> getCandidateInstants(HoodieActiveTimeline activeTimeline,
                                                List<StoragePath> instants,
                                                String currentInstantTime,
                                                long maxAllowableHeartbeatIntervalInMs,
                                                HoodieStorage storage,
                                                String basePath)

Get Candidate Instant to do conflict checking: 1. Skip current writer related instant(currentInstantTime) 2. Skip all instants after currentInstantTime 3. Skip dead writers related instants based on heart-beat 4. Skip pending compaction instant (For now we don' do early conflict check with compact action) Because we don't want to let pending compaction block common writer.

Parameters:

instants -

Returns:

makerToPartitionAndFileID

public static String makerToPartitionAndFileID(String marker)

Get fileID from full marker path, for example: 20210623/0/20210825/932a86d9-5c1d-44c7-ac99-cb88b8ef8478-0_85-15-1390_20220620181735781.parquet.marker.MERGE ==> get 20210623/0/20210825/932a86d9-5c1d-44c7-ac99-cb88b8ef8478-0

Parameters:

marker -

Returns:

markerDirToInstantTime

public static String markerDirToInstantTime(String marker)

Get instantTime from full marker path, for example: /var/folders/t3/th1dw75d0yz2x2k2qt6ys9zh0000gp/T/junit6502909693741900820/dataset/.hoodie/.temp/003 ==> 003

Parameters:

marker -

Returns: