org.apache.hudi.common.table

Class HoodieTableConfig

java.lang.Object

org.apache.hudi.common.config.HoodieConfig

org.apache.hudi.common.table.HoodieTableConfig

All Implemented Interfaces:

Serializable

@Immutable
public class HoodieTableConfig
extends HoodieConfig

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static ConfigProperty<String>	ARCHIVELOG_FOLDER
static ConfigProperty<HoodieFileFormat>	BASE_FILE_FORMAT
static ConfigProperty<String>	BOOTSTRAP_BASE_PATH
static ConfigProperty<String>	BOOTSTRAP_INDEX_CLASS_NAME
static ConfigProperty<Boolean>	BOOTSTRAP_INDEX_ENABLE
static ConfigProperty<Boolean>	CDC_ENABLED
static ConfigProperty<String>	CDC_SUPPLEMENTAL_LOGGING_MODE
static ConfigProperty<String>	CREATE_SCHEMA
static ConfigProperty<String>	DATABASE_NAME
static String	DEFAULT_ARCHIVELOG_FOLDER Deprecated. Use ARCHIVELOG_FOLDER and its methods.
static HoodieFileFormat	DEFAULT_BASE_FILE_FORMAT Deprecated. Use BASE_FILE_FORMAT and its methods.
static String	DEFAULT_BOOTSTRAP_INDEX_CLASS Deprecated. Use BOOTSTRAP_INDEX_CLASS_NAME and its methods.
static HoodieFileFormat	DEFAULT_LOG_FILE_FORMAT Deprecated. Use LOG_FILE_FORMAT and its methods.
static String	DEFAULT_PAYLOAD_CLASS Deprecated. Use PAYLOAD_CLASS_NAME and its methods.
static HoodieTableType	DEFAULT_TABLE_TYPE Deprecated. Use TYPE and its methods.
static HoodieTableVersion	DEFAULT_TABLE_VERSION Deprecated. Use VERSION and its methods.
static ConfigProperty<Boolean>	DROP_PARTITION_COLUMNS
static ConfigProperty<String>	HIVE_STYLE_PARTITIONING_ENABLE
static String	HOODIE_ARCHIVELOG_FOLDER_PROP_NAME Deprecated. Use ARCHIVELOG_FOLDER and its methods.
static String	HOODIE_BASE_FILE_FORMAT_PROP_NAME Deprecated. Use BASE_FILE_FORMAT and its methods.
static String	HOODIE_BOOTSTRAP_BASE_PATH Deprecated. Use BOOTSTRAP_BASE_PATH and its methods.
static String	HOODIE_BOOTSTRAP_INDEX_CLASS_PROP_NAME Deprecated. Use BOOTSTRAP_INDEX_CLASS_NAME and its methods.
static String	HOODIE_LOG_FILE_FORMAT_PROP_NAME Deprecated. Use LOG_FILE_FORMAT and its methods.
static String	HOODIE_PAYLOAD_CLASS_PROP_NAME Deprecated. Use PAYLOAD_CLASS_NAME and its methods.
static String	HOODIE_PROPERTIES_FILE
static String	HOODIE_PROPERTIES_FILE_BACKUP
static String	HOODIE_RO_FILE_FORMAT_PROP_NAME Deprecated. Use BASE_FILE_FORMAT and its methods.
static String	HOODIE_RT_FILE_FORMAT_PROP_NAME Deprecated. Use LOG_FILE_FORMAT and its methods.
static String	HOODIE_TABLE_NAME_KEY
static String	HOODIE_TABLE_NAME_PROP_NAME Deprecated. Use NAME and its methods.
static String	HOODIE_TABLE_PRECOMBINE_FIELD Deprecated. Use PRECOMBINE_FIELD and its methods.
static String	HOODIE_TABLE_TYPE_PROP_NAME Deprecated. Use TYPE and its methods.
static String	HOODIE_TABLE_VERSION_PROP_NAME Deprecated. Use VERSION and its methods.
static String	HOODIE_TIMELINE_LAYOUT_VERSION Deprecated. Use TIMELINE_LAYOUT_VERSION and its methods.
static String	HOODIE_WRITE_TABLE_NAME_KEY
static ConfigProperty<String>	KEY_GENERATOR_CLASS_NAME
static ConfigProperty<HoodieFileFormat>	LOG_FILE_FORMAT
static ConfigProperty<String>	NAME
static String	NO_OP_BOOTSTRAP_INDEX_CLASS
static ConfigProperty<String>	PARTITION_FIELDS
static ConfigProperty<Boolean>	PARTITION_METAFILE_USE_BASE_FORMAT
static ConfigProperty<String>	PAYLOAD_CLASS_NAME
static List<ConfigProperty<String>>	PERSISTED_CONFIG_LIST
static ConfigProperty<Boolean>	POPULATE_META_FIELDS
static ConfigProperty<String>	PRECOMBINE_FIELD
static ConfigProperty<String>	RECORD_MERGER_STRATEGY
static ConfigProperty<String>	RECORDKEY_FIELDS
static ConfigProperty<String>	SECONDARY_INDEXES_METADATA
static ConfigProperty<String>	TABLE_CHECKSUM
static ConfigProperty<String>	TABLE_METADATA_PARTITIONS
static ConfigProperty<String>	TABLE_METADATA_PARTITIONS_INFLIGHT
static ConfigProperty<String>	TIMELINE_LAYOUT_VERSION
static ConfigProperty<HoodieTimelineTimeZone>	TIMELINE_TIMEZONE
static ConfigProperty<HoodieTableType>	TYPE
static ConfigProperty<String>	URL_ENCODE_PARTITIONING
static ConfigProperty<HoodieTableVersion>	VERSION

Fields inherited from class org.apache.hudi.common.config.HoodieConfig

CONFIG_VALUES_DELIMITER, props

Constructor Summary

Constructors

Constructor and Description
HoodieTableConfig() For serializing and de-serializing.
HoodieTableConfig(HoodieStorage storage, StoragePath metaPath, String payloadClassName, String recordMergerStrategyId)

Method Summary

Modifier and Type	Method and Description
HoodieCDCSupplementalLoggingMode	cdcSupplementalLoggingMode()
void	clearMetadataPartitions(HoodieTableMetaClient metaClient) Clear TABLE_METADATA_PARTITIONS TABLE_METADATA_PARTITIONS_INFLIGHT.
static void	create(HoodieStorage storage, StoragePath metadataFolder, Properties properties) Initialize the hoodie meta directory and any necessary files inside the meta (including the hoodie.properties).
static void	delete(HoodieStorage storage, StoragePath metadataFolder, Set<String> deletedProps)
static TypedProperties	fetchConfigs(HoodieStorage storage, String metaPath)
static long	generateChecksum(Properties props)
String	getArchivelogFolder() Get the relative path of archive log folder under metafolder, for this table.
HoodieFileFormat	getBaseFileFormat() Get the base file storage format.
Option<String>	getBootstrapBasePath()
String	getBootstrapIndexClass() Read the payload class for HoodieRecords from the table properties.
String	getDatabaseName() Read the database name.
static String	getDefaultBootstrapIndexClass(Properties props)
String	getHiveStylePartitioningEnable()
String	getKeyGeneratorClassName()
HoodieFileFormat	getLogFileFormat() Get the log Storage Format.
Set<String>	getMetadataPartitions()
Set<String>	getMetadataPartitionsInflight()
String	getPartitionFieldProp() Deprecated. please use getPartitionFields() instead
Option<String[]>	getPartitionFields()
Option<HoodieFileFormat>	getPartitionMetafileFormat() Returns the format to use for partition meta files.
String	getPayloadClass() Read the payload class for HoodieRecords from the table properties.
String	getPreCombineField()
String	getRawRecordKeyFieldProp()
String	getRecordKeyFieldProp()
Option<String[]>	getRecordKeyFields()
String	getRecordMergerStrategy() Read the payload class for HoodieRecords from the table properties.
Option<String>	getSecondaryIndexesMetadata()
Option<org.apache.avro.Schema>	getTableCreateSchema()
String	getTableName() Read the table name.
HoodieTableType	getTableType() Read the table type from the table properties and if not found, return the default.
HoodieTableVersion	getTableVersion()
Option<TimelineLayoutVersion>	getTimelineLayoutVersion()
HoodieTimelineTimeZone	getTimelineTimezone()
String	getUrlEncodePartitioning()
boolean	isCDCEnabled()
boolean	isMetadataPartitionAvailable(MetadataPartitionType partition) Checks if metadata table is enabled and the specified partition has been initialized.
boolean	isMetadataTableAvailable()
boolean	isTablePartitioned()
boolean	populateMetaFields()
Map<String,String>	propsMap()
static void	recover(HoodieStorage fs, StoragePath metadataFolder)
void	setMetadataPartitionsInflight(HoodieTableMetaClient metaClient, List<MetadataPartitionType> partitionTypes) Enables the specified metadata table partition as inflight.
void	setMetadataPartitionsInflight(HoodieTableMetaClient metaClient, MetadataPartitionType... partitionTypes)
void	setMetadataPartitionState(HoodieTableMetaClient metaClient, MetadataPartitionType partitionType, boolean enabled) Enables or disables the specified metadata table partition.
void	setTableVersion(HoodieTableVersion tableVersion)
Boolean	shouldDropPartitionColumns()
static void	update(HoodieStorage storage, StoragePath metadataFolder, Properties updatedProps) Upserts the table config with the set of properties passed in.
static boolean	validateChecksum(Properties props)

Methods inherited from class org.apache.hudi.common.config.HoodieConfig

contains, contains, getBoolean, getBooleanOrDefault, getBooleanOrDefault, getBooleanOrDefault, getDouble, getDoubleOrDefault, getFloat, getFloatOrDefault, getInt, getIntOrDefault, getLong, getLongOrDefault, getProps, getProps, getSplitStrings, getSplitStrings, getString, getString, getStringOrDefault, getStringOrDefault, getStringOrThrow, setAll, setDefault, setDefaultOnCondition, setDefaults, setDefaultValue, setDefaultValue, setValue, setValue

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

HOODIE_PROPERTIES_FILE

public static final String HOODIE_PROPERTIES_FILE

See Also:

Constant Field Values

HOODIE_PROPERTIES_FILE_BACKUP

public static final String HOODIE_PROPERTIES_FILE_BACKUP

See Also:

Constant Field Values

HOODIE_WRITE_TABLE_NAME_KEY

public static final String HOODIE_WRITE_TABLE_NAME_KEY

See Also:

Constant Field Values

HOODIE_TABLE_NAME_KEY

public static final String HOODIE_TABLE_NAME_KEY

See Also:

Constant Field Values

DATABASE_NAME

public static final ConfigProperty<String> DATABASE_NAME

NAME

public static final ConfigProperty<String> NAME

TYPE

public static final ConfigProperty<HoodieTableType> TYPE

VERSION

public static final ConfigProperty<HoodieTableVersion> VERSION

PRECOMBINE_FIELD

public static final ConfigProperty<String> PRECOMBINE_FIELD

PARTITION_FIELDS

public static final ConfigProperty<String> PARTITION_FIELDS

RECORDKEY_FIELDS

public static final ConfigProperty<String> RECORDKEY_FIELDS

CDC_ENABLED

public static final ConfigProperty<Boolean> CDC_ENABLED

CDC_SUPPLEMENTAL_LOGGING_MODE

public static final ConfigProperty<String> CDC_SUPPLEMENTAL_LOGGING_MODE

CREATE_SCHEMA

public static final ConfigProperty<String> CREATE_SCHEMA

BASE_FILE_FORMAT

public static final ConfigProperty<HoodieFileFormat> BASE_FILE_FORMAT

LOG_FILE_FORMAT

public static final ConfigProperty<HoodieFileFormat> LOG_FILE_FORMAT

TIMELINE_LAYOUT_VERSION

public static final ConfigProperty<String> TIMELINE_LAYOUT_VERSION

PAYLOAD_CLASS_NAME

public static final ConfigProperty<String> PAYLOAD_CLASS_NAME

RECORD_MERGER_STRATEGY

public static final ConfigProperty<String> RECORD_MERGER_STRATEGY

ARCHIVELOG_FOLDER

public static final ConfigProperty<String> ARCHIVELOG_FOLDER

BOOTSTRAP_INDEX_ENABLE

public static final ConfigProperty<Boolean> BOOTSTRAP_INDEX_ENABLE

BOOTSTRAP_INDEX_CLASS_NAME

public static final ConfigProperty<String> BOOTSTRAP_INDEX_CLASS_NAME

BOOTSTRAP_BASE_PATH

public static final ConfigProperty<String> BOOTSTRAP_BASE_PATH

POPULATE_META_FIELDS

public static final ConfigProperty<Boolean> POPULATE_META_FIELDS

KEY_GENERATOR_CLASS_NAME

public static final ConfigProperty<String> KEY_GENERATOR_CLASS_NAME

TIMELINE_TIMEZONE

public static final ConfigProperty<HoodieTimelineTimeZone> TIMELINE_TIMEZONE

PARTITION_METAFILE_USE_BASE_FORMAT

public static final ConfigProperty<Boolean> PARTITION_METAFILE_USE_BASE_FORMAT

DROP_PARTITION_COLUMNS

public static final ConfigProperty<Boolean> DROP_PARTITION_COLUMNS

URL_ENCODE_PARTITIONING

public static final ConfigProperty<String> URL_ENCODE_PARTITIONING

HIVE_STYLE_PARTITIONING_ENABLE

public static final ConfigProperty<String> HIVE_STYLE_PARTITIONING_ENABLE

PERSISTED_CONFIG_LIST

public static final List<ConfigProperty<String>> PERSISTED_CONFIG_LIST

NO_OP_BOOTSTRAP_INDEX_CLASS

public static final String NO_OP_BOOTSTRAP_INDEX_CLASS

TABLE_CHECKSUM

public static final ConfigProperty<String> TABLE_CHECKSUM

TABLE_METADATA_PARTITIONS_INFLIGHT

public static final ConfigProperty<String> TABLE_METADATA_PARTITIONS_INFLIGHT

TABLE_METADATA_PARTITIONS

public static final ConfigProperty<String> TABLE_METADATA_PARTITIONS

SECONDARY_INDEXES_METADATA

public static final ConfigProperty<String> SECONDARY_INDEXES_METADATA

HOODIE_RO_FILE_FORMAT_PROP_NAME

@Deprecated
public static final String HOODIE_RO_FILE_FORMAT_PROP_NAME

Deprecated. Use BASE_FILE_FORMAT and its methods.

See Also:

Constant Field Values

HOODIE_RT_FILE_FORMAT_PROP_NAME

@Deprecated
public static final String HOODIE_RT_FILE_FORMAT_PROP_NAME

Deprecated. Use LOG_FILE_FORMAT and its methods.

See Also:

Constant Field Values

HOODIE_TABLE_NAME_PROP_NAME

@Deprecated
public static final String HOODIE_TABLE_NAME_PROP_NAME

Deprecated. Use NAME and its methods.

HOODIE_TABLE_TYPE_PROP_NAME

@Deprecated
public static final String HOODIE_TABLE_TYPE_PROP_NAME

Deprecated. Use TYPE and its methods.

HOODIE_TABLE_VERSION_PROP_NAME

@Deprecated
public static final String HOODIE_TABLE_VERSION_PROP_NAME

Deprecated. Use VERSION and its methods.

HOODIE_TABLE_PRECOMBINE_FIELD

@Deprecated
public static final String HOODIE_TABLE_PRECOMBINE_FIELD

Deprecated. Use PRECOMBINE_FIELD and its methods.

HOODIE_BASE_FILE_FORMAT_PROP_NAME

@Deprecated
public static final String HOODIE_BASE_FILE_FORMAT_PROP_NAME

Deprecated. Use BASE_FILE_FORMAT and its methods.

HOODIE_LOG_FILE_FORMAT_PROP_NAME

@Deprecated
public static final String HOODIE_LOG_FILE_FORMAT_PROP_NAME

Deprecated. Use LOG_FILE_FORMAT and its methods.

HOODIE_TIMELINE_LAYOUT_VERSION

@Deprecated
public static final String HOODIE_TIMELINE_LAYOUT_VERSION

Deprecated. Use TIMELINE_LAYOUT_VERSION and its methods.

HOODIE_PAYLOAD_CLASS_PROP_NAME

@Deprecated
public static final String HOODIE_PAYLOAD_CLASS_PROP_NAME

Deprecated. Use PAYLOAD_CLASS_NAME and its methods.

HOODIE_ARCHIVELOG_FOLDER_PROP_NAME

@Deprecated
public static final String HOODIE_ARCHIVELOG_FOLDER_PROP_NAME

Deprecated. Use ARCHIVELOG_FOLDER and its methods.

HOODIE_BOOTSTRAP_INDEX_CLASS_PROP_NAME

@Deprecated
public static final String HOODIE_BOOTSTRAP_INDEX_CLASS_PROP_NAME

Deprecated. Use BOOTSTRAP_INDEX_CLASS_NAME and its methods.

HOODIE_BOOTSTRAP_BASE_PATH

@Deprecated
public static final String HOODIE_BOOTSTRAP_BASE_PATH

Deprecated. Use BOOTSTRAP_BASE_PATH and its methods.

DEFAULT_TABLE_TYPE

@Deprecated
public static final HoodieTableType DEFAULT_TABLE_TYPE

Deprecated. Use TYPE and its methods.

DEFAULT_TABLE_VERSION

@Deprecated
public static final HoodieTableVersion DEFAULT_TABLE_VERSION

Deprecated. Use VERSION and its methods.

DEFAULT_BASE_FILE_FORMAT

@Deprecated
public static final HoodieFileFormat DEFAULT_BASE_FILE_FORMAT

Deprecated. Use BASE_FILE_FORMAT and its methods.

DEFAULT_LOG_FILE_FORMAT

@Deprecated
public static final HoodieFileFormat DEFAULT_LOG_FILE_FORMAT

Deprecated. Use LOG_FILE_FORMAT and its methods.

DEFAULT_PAYLOAD_CLASS

@Deprecated
public static final String DEFAULT_PAYLOAD_CLASS

Deprecated. Use PAYLOAD_CLASS_NAME and its methods.

DEFAULT_BOOTSTRAP_INDEX_CLASS

@Deprecated
public static final String DEFAULT_BOOTSTRAP_INDEX_CLASS

Deprecated. Use BOOTSTRAP_INDEX_CLASS_NAME and its methods.

DEFAULT_ARCHIVELOG_FOLDER

@Deprecated
public static final String DEFAULT_ARCHIVELOG_FOLDER

Deprecated. Use ARCHIVELOG_FOLDER and its methods.

Constructor Detail

HoodieTableConfig

public HoodieTableConfig(HoodieStorage storage,
                         StoragePath metaPath,
                         String payloadClassName,
                         String recordMergerStrategyId)

HoodieTableConfig

public HoodieTableConfig()

For serializing and de-serializing.

Method Detail

fetchConfigs

public static TypedProperties fetchConfigs(HoodieStorage storage,
                                           String metaPath)
                                    throws IOException

Throws:

IOException

recover

public static void recover(HoodieStorage fs,
                           StoragePath metadataFolder)
                    throws IOException

Throws:

IOException

update

public static void update(HoodieStorage storage,
                          StoragePath metadataFolder,
                          Properties updatedProps)

Upserts the table config with the set of properties passed in. We implement a fail-safe backup protocol here for safely updating with recovery and also ensuring the table config continues to be readable.

delete

public static void delete(HoodieStorage storage,
                          StoragePath metadataFolder,
                          Set<String> deletedProps)

create

public static void create(HoodieStorage storage,
                          StoragePath metadataFolder,
                          Properties properties)
                   throws IOException

Initialize the hoodie meta directory and any necessary files inside the meta (including the hoodie.properties).

Throws:

IOException

generateChecksum

public static long generateChecksum(Properties props)

validateChecksum

public static boolean validateChecksum(Properties props)

getTableType

public HoodieTableType getTableType()

Read the table type from the table properties and if not found, return the default.

getTimelineLayoutVersion

public Option<TimelineLayoutVersion> getTimelineLayoutVersion()

getTableVersion

public HoodieTableVersion getTableVersion()

Returns:

the hoodie.table.version from hoodie.properties file.

setTableVersion

public void setTableVersion(HoodieTableVersion tableVersion)

getPayloadClass

public String getPayloadClass()

Read the payload class for HoodieRecords from the table properties.

getRecordMergerStrategy

public String getRecordMergerStrategy()

Read the payload class for HoodieRecords from the table properties.

getPreCombineField

public String getPreCombineField()

getRecordKeyFields

public Option<String[]> getRecordKeyFields()

getPartitionFields

public Option<String[]> getPartitionFields()

isTablePartitioned

public boolean isTablePartitioned()

getSecondaryIndexesMetadata

public Option<String> getSecondaryIndexesMetadata()

getPartitionFieldProp

@Deprecated
public String getPartitionFieldProp()

Deprecated. please use getPartitionFields() instead

getBootstrapIndexClass

public String getBootstrapIndexClass()

Read the payload class for HoodieRecords from the table properties.

getDefaultBootstrapIndexClass

public static String getDefaultBootstrapIndexClass(Properties props)

getBootstrapBasePath

public Option<String> getBootstrapBasePath()

getTableCreateSchema

public Option<org.apache.avro.Schema> getTableCreateSchema()

getDatabaseName

public String getDatabaseName()

Read the database name.

getTableName

public String getTableName()

Read the table name.

getBaseFileFormat

public HoodieFileFormat getBaseFileFormat()

Get the base file storage format.

Returns:

HoodieFileFormat for the base file Storage format

getLogFileFormat

public HoodieFileFormat getLogFileFormat()

Get the log Storage Format.

Returns:

HoodieFileFormat for the log Storage format

getArchivelogFolder

public String getArchivelogFolder()

Get the relative path of archive log folder under metafolder, for this table.

populateMetaFields

public boolean populateMetaFields()

getRecordKeyFieldProp

public String getRecordKeyFieldProp()

getRawRecordKeyFieldProp

public String getRawRecordKeyFieldProp()

isCDCEnabled

public boolean isCDCEnabled()

cdcSupplementalLoggingMode

public HoodieCDCSupplementalLoggingMode cdcSupplementalLoggingMode()

getKeyGeneratorClassName

public String getKeyGeneratorClassName()

getTimelineTimezone

public HoodieTimelineTimeZone getTimelineTimezone()

getHiveStylePartitioningEnable

public String getHiveStylePartitioningEnable()

getUrlEncodePartitioning

public String getUrlEncodePartitioning()

shouldDropPartitionColumns

public Boolean shouldDropPartitionColumns()

getMetadataPartitionsInflight

public Set<String> getMetadataPartitionsInflight()

getMetadataPartitions

public Set<String> getMetadataPartitions()

isMetadataTableAvailable

public boolean isMetadataTableAvailable()

isMetadataPartitionAvailable

public boolean isMetadataPartitionAvailable(MetadataPartitionType partition)

Checks if metadata table is enabled and the specified partition has been initialized.

Parameters:

partition - The partition to check

setMetadataPartitionState

public void setMetadataPartitionState(HoodieTableMetaClient metaClient,
                                      MetadataPartitionType partitionType,
                                      boolean enabled)

Enables or disables the specified metadata table partition.

Parameters:

partitionType - The partition

enabled - If true, the partition is enabled, else disabled

setMetadataPartitionsInflight

public void setMetadataPartitionsInflight(HoodieTableMetaClient metaClient,
                                          List<MetadataPartitionType> partitionTypes)

Enables the specified metadata table partition as inflight.

Parameters:

partitionTypes - The list of partitions to enable as inflight.

setMetadataPartitionsInflight

public void setMetadataPartitionsInflight(HoodieTableMetaClient metaClient,
                                          MetadataPartitionType... partitionTypes)

clearMetadataPartitions

public void clearMetadataPartitions(HoodieTableMetaClient metaClient)

Clear TABLE_METADATA_PARTITIONS TABLE_METADATA_PARTITIONS_INFLIGHT.

getPartitionMetafileFormat

public Option<HoodieFileFormat> getPartitionMetafileFormat()

Returns the format to use for partition meta files.

propsMap

public Map<String,String> propsMap()