org.apache.hudi.common.model

Class DefaultHoodieRecordPayload

java.lang.Object

org.apache.hudi.common.model.BaseAvroPayload

org.apache.hudi.common.model.OverwriteWithLatestAvroPayload

org.apache.hudi.common.model.DefaultHoodieRecordPayload

All Implemented Interfaces:

Serializable, HoodieRecordPayload<OverwriteWithLatestAvroPayload>

Direct Known Subclasses:

EventTimeAvroPayload, FirstValueAvroPayload

public class DefaultHoodieRecordPayload
extends OverwriteWithLatestAvroPayload

HoodieRecordPayload impl that honors ordering field in both preCombine and combineAndGetUpdateValue.

1. preCombine - Picks the latest delta record for a key, based on an ordering field 2. combineAndGetUpdateValue/getInsertValue - Chooses the latest record based on ordering field value.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static String	DELETE_KEY
static String	DELETE_MARKER
static String	METADATA_EVENT_TIME_KEY

Fields inherited from class org.apache.hudi.common.model.BaseAvroPayload

isDeletedRecord, orderingVal, recordBytes

Constructor Summary

Constructors

Constructor and Description
DefaultHoodieRecordPayload(org.apache.avro.generic.GenericRecord record, Comparable orderingVal)
DefaultHoodieRecordPayload(Option<org.apache.avro.generic.GenericRecord> record)

Method Summary

Modifier and Type	Method and Description
Option<org.apache.avro.generic.IndexedRecord>	combineAndGetUpdateValue(org.apache.avro.generic.IndexedRecord currentValue, org.apache.avro.Schema schema, Properties properties) This methods lets you write custom merging/combining logic to produce new values as a function of current value on storage and whats contained in this object.
Option<org.apache.avro.generic.IndexedRecord>	getInsertValue(org.apache.avro.Schema schema, Properties properties) Generates an avro record out of the given HoodieRecordPayload, to be written out to storage.
Option<Map<String,String>>	getMetadata() This method can be used to extract some metadata from HoodieRecordPayload.
boolean	isDeleted(org.apache.avro.Schema schema, Properties props) Defines whether this implementation of HoodieRecordPayload is deleted.
protected boolean	isDeleteRecord(org.apache.avro.generic.GenericRecord genericRecord, Properties properties)
protected boolean	needUpdatingPersistedRecord(org.apache.avro.generic.IndexedRecord currentValue, org.apache.avro.generic.IndexedRecord incomingRecord, Properties properties)

Methods inherited from class org.apache.hudi.common.model.OverwriteWithLatestAvroPayload

combineAndGetUpdateValue, getInsertValue, getOrderingValue, overwriteField, preCombine

Methods inherited from class org.apache.hudi.common.model.BaseAvroPayload

canProduceSentinel, getOrderingVal, isDeleteRecord

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hudi.common.model.HoodieRecordPayload

preCombine, preCombine

Field Detail

METADATA_EVENT_TIME_KEY

public static final String METADATA_EVENT_TIME_KEY

See Also:

Constant Field Values

DELETE_KEY

public static final String DELETE_KEY

See Also:

Constant Field Values

DELETE_MARKER

public static final String DELETE_MARKER

See Also:

Constant Field Values

Constructor Detail

DefaultHoodieRecordPayload

public DefaultHoodieRecordPayload(org.apache.avro.generic.GenericRecord record,
                                  Comparable orderingVal)

DefaultHoodieRecordPayload

public DefaultHoodieRecordPayload(Option<org.apache.avro.generic.GenericRecord> record)

Method Detail

combineAndGetUpdateValue

public Option<org.apache.avro.generic.IndexedRecord> combineAndGetUpdateValue(org.apache.avro.generic.IndexedRecord currentValue,
                                                                              org.apache.avro.Schema schema,
                                                                              Properties properties)
                                                                       throws IOException

Description copied from interface: HoodieRecordPayload

This methods lets you write custom merging/combining logic to produce new values as a function of current value on storage and whats contained in this object. Implementations can leverage properties if required.

eg: 1) You are updating counters, you may want to add counts to currentValue and write back updated counts 2) You may be reading DB redo logs, and merge them with current image for a database row on storage

Parameters:

currentValue - Current value in storage, to merge/combine this payload with

schema - Schema used for record

properties - Payload related properties. For example pass the ordering field(s) name to extract from value in storage.

Returns:

new combined/merged value to be written back to storage. EMPTY to skip writing this record.

Throws:

IOException

getInsertValue

public Option<org.apache.avro.generic.IndexedRecord> getInsertValue(org.apache.avro.Schema schema,
                                                                    Properties properties)
                                                             throws IOException

Description copied from interface: HoodieRecordPayload

Generates an avro record out of the given HoodieRecordPayload, to be written out to storage. Called when writing a new value for the given HoodieKey, wherein there is no existing record in storage to be combined against. (i.e insert) Return EMPTY to skip writing this record. Implementations can leverage properties if required.

Parameters:

schema - Schema used for record

properties - Payload related properties. For example pass the ordering field(s) name to extract from value in storage.

Returns:

the IndexedRecord to be inserted.

Throws:

IOException

isDeleted

public boolean isDeleted(org.apache.avro.Schema schema,
                         Properties props)

Description copied from class: BaseAvroPayload

Defines whether this implementation of HoodieRecordPayload is deleted. We will not do deserialization in this method.

Overrides:

isDeleted in class BaseAvroPayload

isDeleteRecord

protected boolean isDeleteRecord(org.apache.avro.generic.GenericRecord genericRecord,
                                 Properties properties)

Parameters:

genericRecord - instance of GenericRecord of interest.

properties - payload related properties

getMetadata

public Option<Map<String,String>> getMetadata()

Description copied from interface: HoodieRecordPayload

This method can be used to extract some metadata from HoodieRecordPayload. The metadata is passed to WriteStatus.markSuccess() and WriteStatus.markFailure() in order to compute some aggregate metrics using the metadata in the context of a write success or failure.

Returns:

the metadata in the form of Map if any.

needUpdatingPersistedRecord

protected boolean needUpdatingPersistedRecord(org.apache.avro.generic.IndexedRecord currentValue,
                                              org.apache.avro.generic.IndexedRecord incomingRecord,
                                              Properties properties)