org.apache.hudi.common.model

Class HoodieRecord<T>

java.lang.Object

org.apache.hudi.common.model.HoodieRecord<T>

All Implemented Interfaces:

com.esotericsoftware.kryo.KryoSerializable, Serializable, HoodieRecordCompatibilityInterface

Direct Known Subclasses:

HoodieAvroIndexedRecord, HoodieAvroRecord, HoodieEmptyRecord

public abstract class HoodieRecord<T>
extends Object
implements HoodieRecordCompatibilityInterface, com.esotericsoftware.kryo.KryoSerializable, Serializable

A Single Record managed by Hoodie.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HoodieRecord.HoodieMetadataField
static class	HoodieRecord.HoodieRecordType

Field Summary

Fields

Modifier and Type	Field and Description
static String	COMMIT_SEQNO_METADATA_FIELD
static int	COMMIT_SEQNO_METADATA_FIELD_ORD
static String	COMMIT_TIME_METADATA_FIELD
static int	COMMIT_TIME_METADATA_FIELD_ORD
protected HoodieRecordLocation	currentLocation Current location of record on storage.
protected T	data Actual payload of the record.
static int	FILENAME_META_FIELD_ORD
static String	FILENAME_METADATA_FIELD
static String	HOODIE_IS_DELETED_FIELD
static List<String>	HOODIE_META_COLUMNS
static Map<String,Integer>	HOODIE_META_COLUMNS_NAME_TO_POS
static Set<String>	HOODIE_META_COLUMNS_WITH_OPERATION
protected boolean	ignoreIndexUpdate If set, not update index after written.
protected HoodieKey	key Identifies the record across the table.
protected Option<Map<String,String>>	metaData The metaData of the record.
protected HoodieRecordLocation	newLocation New location of record on storage, after written.
protected HoodieOperation	operation The cdc operation.
static String	OPERATION_METADATA_FIELD
static int	PARTITION_PATH_META_FIELD_ORD
static String	PARTITION_PATH_METADATA_FIELD
static int	RECORD_KEY_META_FIELD_ORD
static String	RECORD_KEY_METADATA_FIELD
static org.apache.hudi.common.model.HoodieRecord.EmptyRecord	SENTINEL A special record returned by HoodieRecordPayload, which means we should just skip this record.

Constructor Summary

Constructors

Constructor and Description
HoodieRecord()
HoodieRecord(HoodieKey key, T data)
HoodieRecord(HoodieKey key, T data, HoodieOperation operation, HoodieRecordLocation currentLocation, HoodieRecordLocation newLocation)
HoodieRecord(HoodieKey key, T data, HoodieOperation operation, Option<Map<String,String>> metaData)
HoodieRecord(HoodieRecord<T> record)

Method Summary

Modifier and Type	Method and Description
void	checkState()
void	clearNewLocation() Clears the new currentLocation of the record.
abstract HoodieRecord<T>	copy() This is used to copy data.
void	deflate() Release the actual payload, to ease memory pressure.
boolean	equals(Object o)
static String	generateSequenceId(String instantTime, int partitionId, long recordIndex)
abstract Object[]	getColumnValues(org.apache.avro.Schema recordSchema, String[] columns, boolean consistentLogicalTimestampEnabled) Get column in record to support RDDCustomColumnsSortPartitioner
HoodieRecordLocation	getCurrentLocation()
T	getData()
boolean	getIgnoreIndexUpdate()
HoodieKey	getKey()
abstract Option<Map<String,String>>	getMetadata()
HoodieRecordLocation	getNewLocation()
HoodieOperation	getOperation()
abstract Comparable<?>	getOrderingValue(org.apache.avro.Schema recordSchema, Properties props)
String	getPartitionPath()
String	getRecordKey()
abstract String	getRecordKey(org.apache.avro.Schema recordSchema, Option<BaseKeyGenerator> keyGeneratorOpt)
abstract String	getRecordKey(org.apache.avro.Schema recordSchema, String keyFieldName)
abstract HoodieRecord.HoodieRecordType	getRecordType()
int	hashCode()
protected static boolean	hasMetaFields(org.apache.avro.Schema schema)
boolean	isCurrentLocationKnown()
abstract boolean	isDelete(org.apache.avro.Schema recordSchema, Properties props)
abstract HoodieRecord	joinWith(HoodieRecord other, org.apache.avro.Schema targetSchema) Support bootstrap.
abstract HoodieRecord<T>	newInstance()
abstract HoodieRecord<T>	newInstance(HoodieKey key)
abstract HoodieRecord<T>	newInstance(HoodieKey key, HoodieOperation op)
abstract HoodieRecord	prependMetaFields(org.apache.avro.Schema recordSchema, org.apache.avro.Schema targetSchema, MetadataValues metadataValues, Properties props) Rewrites record into new target schema containing Hudi-specific meta-fields NOTE: This operation is idempotent
void	read(com.esotericsoftware.kryo.Kryo kryo, com.esotericsoftware.kryo.io.Input input) NOTE: This method is declared final to make sure there's no polymorphism and therefore JIT compiler could perform more aggressive optimizations
protected abstract T	readRecordPayload(com.esotericsoftware.kryo.Kryo kryo, com.esotericsoftware.kryo.io.Input input)
HoodieRecord	rewriteRecordWithNewSchema(org.apache.avro.Schema recordSchema, Properties props, org.apache.avro.Schema newSchema)
abstract HoodieRecord	rewriteRecordWithNewSchema(org.apache.avro.Schema recordSchema, Properties props, org.apache.avro.Schema newSchema, Map<String,String> renameCols) Support schema evolution.
void	seal()
HoodieRecord	setCurrentLocation(HoodieRecordLocation location) Sets the current currentLocation of the record.
void	setIgnoreIndexUpdate(boolean ignoreFlag) Sets the ignore flag.
void	setNewLocation(HoodieRecordLocation location) Sets the new currentLocation of the record, after being written.
abstract boolean	shouldIgnore(org.apache.avro.Schema recordSchema, Properties props) Is EmptyRecord.
String	toString()
void	unseal()
void	write(com.esotericsoftware.kryo.Kryo kryo, com.esotericsoftware.kryo.io.Output output) NOTE: This method is declared final to make sure there's no polymorphism and therefore JIT compiler could perform more aggressive optimizations
protected abstract void	writeRecordPayload(T payload, com.esotericsoftware.kryo.Kryo kryo, com.esotericsoftware.kryo.io.Output output)

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hudi.common.model.HoodieRecordCompatibilityInterface

toIndexedRecord, truncateRecordKey, wrapIntoHoodieRecordPayloadWithKeyGen, wrapIntoHoodieRecordPayloadWithParams

Field Detail

COMMIT_TIME_METADATA_FIELD

public static final String COMMIT_TIME_METADATA_FIELD

COMMIT_SEQNO_METADATA_FIELD

public static final String COMMIT_SEQNO_METADATA_FIELD

RECORD_KEY_METADATA_FIELD

public static final String RECORD_KEY_METADATA_FIELD

PARTITION_PATH_METADATA_FIELD

public static final String PARTITION_PATH_METADATA_FIELD

FILENAME_METADATA_FIELD

public static final String FILENAME_METADATA_FIELD

OPERATION_METADATA_FIELD

public static final String OPERATION_METADATA_FIELD

HOODIE_IS_DELETED_FIELD

public static final String HOODIE_IS_DELETED_FIELD

See Also:

Constant Field Values

SENTINEL

public static final org.apache.hudi.common.model.HoodieRecord.EmptyRecord SENTINEL

A special record returned by HoodieRecordPayload, which means we should just skip this record. This record is only used for HoodieRecordPayload currently, so it should not shuffle though network, we can compare the record locally by the equal method. The HoodieRecordPayload#combineAndGetUpdateValue and HoodieRecordPayload#getInsertValue have 3 kind of return: 1、Option.empty This means we should delete this record. 2、IGNORE_RECORD This means we should not process this record,just skip. 3、Other non-empty record This means we should process this record. We can see the usage of IGNORE_RECORD in org.apache.spark.sql.hudi.command.payload.ExpressionPayload

HOODIE_META_COLUMNS

public static final List<String> HOODIE_META_COLUMNS

HOODIE_META_COLUMNS_WITH_OPERATION

public static final Set<String> HOODIE_META_COLUMNS_WITH_OPERATION

HOODIE_META_COLUMNS_NAME_TO_POS

public static final Map<String,Integer> HOODIE_META_COLUMNS_NAME_TO_POS

RECORD_KEY_META_FIELD_ORD

public static int RECORD_KEY_META_FIELD_ORD

PARTITION_PATH_META_FIELD_ORD

public static int PARTITION_PATH_META_FIELD_ORD

FILENAME_META_FIELD_ORD

public static int FILENAME_META_FIELD_ORD

COMMIT_TIME_METADATA_FIELD_ORD

public static int COMMIT_TIME_METADATA_FIELD_ORD

COMMIT_SEQNO_METADATA_FIELD_ORD

public static int COMMIT_SEQNO_METADATA_FIELD_ORD

key

protected HoodieKey key

Identifies the record across the table.

data

protected T data

Actual payload of the record.

currentLocation

protected HoodieRecordLocation currentLocation

Current location of record on storage. Filled in by looking up index

newLocation

protected HoodieRecordLocation newLocation

New location of record on storage, after written.

ignoreIndexUpdate

protected boolean ignoreIndexUpdate

If set, not update index after written.

operation

protected HoodieOperation operation

The cdc operation.

metaData

protected Option<Map<String,String>> metaData

The metaData of the record.

Constructor Detail

HoodieRecord

public HoodieRecord(HoodieKey key,
                    T data)

HoodieRecord

public HoodieRecord(HoodieKey key,
                    T data,
                    HoodieOperation operation,
                    Option<Map<String,String>> metaData)

HoodieRecord

public HoodieRecord(HoodieKey key,
                    T data,
                    HoodieOperation operation,
                    HoodieRecordLocation currentLocation,
                    HoodieRecordLocation newLocation)

HoodieRecord

public HoodieRecord(HoodieRecord<T> record)

HoodieRecord

public HoodieRecord()

Method Detail

newInstance

public abstract HoodieRecord<T> newInstance()

newInstance

public abstract HoodieRecord<T> newInstance(HoodieKey key,
                                            HoodieOperation op)

newInstance

public abstract HoodieRecord<T> newInstance(HoodieKey key)

getKey

public HoodieKey getKey()

getOperation

public HoodieOperation getOperation()

getOrderingValue

public abstract Comparable<?> getOrderingValue(org.apache.avro.Schema recordSchema,
                                               Properties props)

getData

public T getData()

deflate

public void deflate()

Release the actual payload, to ease memory pressure. To be called after the record has been written to storage. Once deflated, cannot be inflated.

setCurrentLocation

public HoodieRecord setCurrentLocation(HoodieRecordLocation location)

Sets the current currentLocation of the record. This should happen exactly-once

getCurrentLocation

@Nullable
public HoodieRecordLocation getCurrentLocation()

setNewLocation

public void setNewLocation(HoodieRecordLocation location)

Sets the new currentLocation of the record, after being written. This again should happen exactly-once.

getNewLocation

@Nullable
public HoodieRecordLocation getNewLocation()

isCurrentLocationKnown

public boolean isCurrentLocationKnown()

setIgnoreIndexUpdate

public void setIgnoreIndexUpdate(boolean ignoreFlag)

Sets the ignore flag.

getIgnoreIndexUpdate

public boolean getIgnoreIndexUpdate()

equals

public boolean equals(Object o)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

toString

public String toString()

Overrides:

toString in class Object

getPartitionPath

public String getPartitionPath()

getRecordKey

public String getRecordKey()

getRecordType

public abstract HoodieRecord.HoodieRecordType getRecordType()

getRecordKey

public abstract String getRecordKey(org.apache.avro.Schema recordSchema,
                                    Option<BaseKeyGenerator> keyGeneratorOpt)

getRecordKey

public abstract String getRecordKey(org.apache.avro.Schema recordSchema,
                                    String keyFieldName)

seal

public void seal()

unseal

public void unseal()

checkState

public void checkState()

writeRecordPayload

protected abstract void writeRecordPayload(T payload,
                                           com.esotericsoftware.kryo.Kryo kryo,
                                           com.esotericsoftware.kryo.io.Output output)

readRecordPayload

protected abstract T readRecordPayload(com.esotericsoftware.kryo.Kryo kryo,
                                       com.esotericsoftware.kryo.io.Input input)

clearNewLocation

public void clearNewLocation()

Clears the new currentLocation of the record. This is required in the delete path so that Index can track that this record was deleted.

write

public final void write(com.esotericsoftware.kryo.Kryo kryo,
                        com.esotericsoftware.kryo.io.Output output)

NOTE: This method is declared final to make sure there's no polymorphism and therefore JIT compiler could perform more aggressive optimizations

Specified by:

write in interface com.esotericsoftware.kryo.KryoSerializable

read

public final void read(com.esotericsoftware.kryo.Kryo kryo,
                       com.esotericsoftware.kryo.io.Input input)

NOTE: This method is declared final to make sure there's no polymorphism and therefore JIT compiler could perform more aggressive optimizations

Specified by:

read in interface com.esotericsoftware.kryo.KryoSerializable

getColumnValues

public abstract Object[] getColumnValues(org.apache.avro.Schema recordSchema,
                                         String[] columns,
                                         boolean consistentLogicalTimestampEnabled)

Get column in record to support RDDCustomColumnsSortPartitioner

Returns:

joinWith

public abstract HoodieRecord joinWith(HoodieRecord other,
                                      org.apache.avro.Schema targetSchema)

Support bootstrap.

prependMetaFields

public abstract HoodieRecord prependMetaFields(org.apache.avro.Schema recordSchema,
                                               org.apache.avro.Schema targetSchema,
                                               MetadataValues metadataValues,
                                               Properties props)

Rewrites record into new target schema containing Hudi-specific meta-fields NOTE: This operation is idempotent

rewriteRecordWithNewSchema

public abstract HoodieRecord rewriteRecordWithNewSchema(org.apache.avro.Schema recordSchema,
                                                        Properties props,
                                                        org.apache.avro.Schema newSchema,
                                                        Map<String,String> renameCols)

Support schema evolution.

rewriteRecordWithNewSchema

public HoodieRecord rewriteRecordWithNewSchema(org.apache.avro.Schema recordSchema,
                                               Properties props,
                                               org.apache.avro.Schema newSchema)

isDelete

public abstract boolean isDelete(org.apache.avro.Schema recordSchema,
                                 Properties props)
                          throws IOException

Throws:

IOException

shouldIgnore

public abstract boolean shouldIgnore(org.apache.avro.Schema recordSchema,
                                     Properties props)
                              throws IOException

Is EmptyRecord. Generated by ExpressionPayload.

Throws:

IOException

copy

public abstract HoodieRecord<T> copy()

This is used to copy data.

getMetadata

public abstract Option<Map<String,String>> getMetadata()

generateSequenceId

public static String generateSequenceId(String instantTime,
                                        int partitionId,
                                        long recordIndex)

hasMetaFields

protected static boolean hasMetaFields(org.apache.avro.Schema schema)