org.apache.hudi.common.util

Class CommitUtils

java.lang.Object

org.apache.hudi.common.util.CommitUtils

public class CommitUtils
extends Object

Helper class to generate commit metadata.

Constructor Summary

Constructors

Constructor and Description
CommitUtils()

Method Summary

Modifier and Type	Method and Description
static HoodieCommitMetadata	buildMetadata(List<HoodieWriteStat> writeStats, Map<String,List<String>> partitionToReplaceFileIds, Option<Map<String,String>> extraMetadata, WriteOperationType operationType, String schemaToStoreInCommit, String commitActionType)
static Set<Pair<String,String>>	flattenPartitionToReplaceFileIds(Map<String,List<String>> partitionToReplaceFileIds)
static String	getCheckpointValueAsString(String identifier, String batchId)
static String	getCommitActionType(HoodieTableType tableType) Gets the commit action type for given table type.
static String	getCommitActionType(WriteOperationType operation, HoodieTableType tableType) Gets the commit action type for given write operation and table type.
static Set<Pair<String,String>>	getPartitionAndFileIdWithoutSuffix(Map<String,List<HoodieWriteStat>> partitionToWriteStats)
static Set<Pair<String,String>>	getPartitionAndFileIdWithoutSuffixFromSpecificRecord(Map<String,List<HoodieWriteStat>> partitionToWriteStats)
static Option<String>	getValidCheckpointForCurrentWriter(HoodieTimeline timeline, String checkpointKey, String keyToLookup) Process previous commits metadata in the timeline to determine the checkpoint given a checkpoint key.
static Option<String>	readCheckpointValue(String value, String id)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CommitUtils

public CommitUtils()

Method Detail

getCommitActionType

public static String getCommitActionType(WriteOperationType operation,
                                         HoodieTableType tableType)

Gets the commit action type for given write operation and table type. Use this API when commit action type can differ not only on the basis of table type but also write operation type. For example, INSERT_OVERWRITE/INSERT_OVERWRITE_TABLE operations have REPLACE commit action type.

getCommitActionType

public static String getCommitActionType(HoodieTableType tableType)

Gets the commit action type for given table type. Note: Use this API only when the commit action type is not dependent on the write operation type. See getCommitActionType(WriteOperationType, HoodieTableType) for more details.

buildMetadata

public static HoodieCommitMetadata buildMetadata(List<HoodieWriteStat> writeStats,
                                                 Map<String,List<String>> partitionToReplaceFileIds,
                                                 Option<Map<String,String>> extraMetadata,
                                                 WriteOperationType operationType,
                                                 String schemaToStoreInCommit,
                                                 String commitActionType)

getPartitionAndFileIdWithoutSuffixFromSpecificRecord

public static Set<Pair<String,String>> getPartitionAndFileIdWithoutSuffixFromSpecificRecord(Map<String,List<HoodieWriteStat>> partitionToWriteStats)

getPartitionAndFileIdWithoutSuffix

public static Set<Pair<String,String>> getPartitionAndFileIdWithoutSuffix(Map<String,List<HoodieWriteStat>> partitionToWriteStats)

flattenPartitionToReplaceFileIds

public static Set<Pair<String,String>> flattenPartitionToReplaceFileIds(Map<String,List<String>> partitionToReplaceFileIds)

getValidCheckpointForCurrentWriter

public static Option<String> getValidCheckpointForCurrentWriter(HoodieTimeline timeline,
                                                                String checkpointKey,
                                                                String keyToLookup)

Process previous commits metadata in the timeline to determine the checkpoint given a checkpoint key. NOTE: This is very similar in intent to DeltaSync#getLatestCommitMetadataWithValidCheckpointInfo except that different deployment models (deltastreamer or spark structured streaming) could have different checkpoint keys.

Parameters:

timeline - completed commits in active timeline.

checkpointKey - the checkpoint key in the extra metadata of the commit.

keyToLookup - key of interest for which checkpoint is looked up for.

Returns:

An optional commit metadata with latest checkpoint.

readCheckpointValue

public static Option<String> readCheckpointValue(String value,
                                                 String id)

getCheckpointValueAsString

public static String getCheckpointValueAsString(String identifier,
                                                String batchId)