org.apache.hudi.common.model

Class AWSDmsAvroPayload

java.lang.Object

org.apache.hudi.common.model.BaseAvroPayload

org.apache.hudi.common.model.OverwriteWithLatestAvroPayload

org.apache.hudi.common.model.AWSDmsAvroPayload

All Implemented Interfaces:

Serializable, HoodieRecordPayload<OverwriteWithLatestAvroPayload>

public class AWSDmsAvroPayload
extends OverwriteWithLatestAvroPayload

Provides support for seamlessly applying changes captured via Amazon Database Migration Service onto S3. Typically, we get the following pattern of full change records corresponding to DML against the source database - Full load records with no `Op` field - For inserts against the source table, records contain full after image with `Op=I` - For updates against the source table, records contain full after image with `Op=U` - For deletes against the source table, records contain full before image with `Op=D` This payload implementation will issue matching insert, delete, updates against the hudi table

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static String	OP_FIELD

Fields inherited from class org.apache.hudi.common.model.BaseAvroPayload

isDeletedRecord, orderingVal, recordBytes

Constructor Summary

Constructors

Constructor and Description
AWSDmsAvroPayload(org.apache.avro.generic.GenericRecord record, Comparable orderingVal)
AWSDmsAvroPayload(Option<org.apache.avro.generic.GenericRecord> record)

Method Summary

Modifier and Type	Method and Description
Option<org.apache.avro.generic.IndexedRecord>	combineAndGetUpdateValue(org.apache.avro.generic.IndexedRecord currentValue, org.apache.avro.Schema schema) This methods is deprecated.
Option<org.apache.avro.generic.IndexedRecord>	combineAndGetUpdateValue(org.apache.avro.generic.IndexedRecord currentValue, org.apache.avro.Schema schema, Properties properties) This methods lets you write custom merging/combining logic to produce new values as a function of current value on storage and whats contained in this object.
Option<org.apache.avro.generic.IndexedRecord>	getInsertValue(org.apache.avro.Schema schema) This method is deprecated.
Option<org.apache.avro.generic.IndexedRecord>	getInsertValue(org.apache.avro.Schema schema, Properties properties) Generates an avro record out of the given HoodieRecordPayload, to be written out to storage.
protected boolean	isDeleteRecord(org.apache.avro.generic.GenericRecord record)

Methods inherited from class org.apache.hudi.common.model.OverwriteWithLatestAvroPayload

getOrderingValue, overwriteField, preCombine

Methods inherited from class org.apache.hudi.common.model.BaseAvroPayload

canProduceSentinel, getOrderingVal, isDeleted

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hudi.common.model.HoodieRecordPayload

getMetadata, preCombine, preCombine

Field Detail

OP_FIELD

public static final String OP_FIELD

See Also:

Constant Field Values

Constructor Detail

AWSDmsAvroPayload

public AWSDmsAvroPayload(org.apache.avro.generic.GenericRecord record,
                         Comparable orderingVal)

AWSDmsAvroPayload

public AWSDmsAvroPayload(Option<org.apache.avro.generic.GenericRecord> record)

Method Detail

getInsertValue

public Option<org.apache.avro.generic.IndexedRecord> getInsertValue(org.apache.avro.Schema schema,
                                                                    Properties properties)
                                                             throws IOException

Description copied from interface: HoodieRecordPayload

Generates an avro record out of the given HoodieRecordPayload, to be written out to storage. Called when writing a new value for the given HoodieKey, wherein there is no existing record in storage to be combined against. (i.e insert) Return EMPTY to skip writing this record. Implementations can leverage properties if required.

Parameters:

schema - Schema used for record

properties - Payload related properties. For example pass the ordering field(s) name to extract from value in storage.

Returns:

the IndexedRecord to be inserted.

Throws:

IOException

getInsertValue

public Option<org.apache.avro.generic.IndexedRecord> getInsertValue(org.apache.avro.Schema schema)
                                                             throws IOException

Description copied from interface: HoodieRecordPayload

This method is deprecated. Refer to HoodieRecordPayload.getInsertValue(Schema, Properties) for java docs.

Specified by:

getInsertValue in interface HoodieRecordPayload<OverwriteWithLatestAvroPayload>

Overrides:

getInsertValue in class OverwriteWithLatestAvroPayload

Parameters:

schema - Schema used for record

Returns:

the IndexedRecord to be inserted.

Throws:

IOException

combineAndGetUpdateValue

public Option<org.apache.avro.generic.IndexedRecord> combineAndGetUpdateValue(org.apache.avro.generic.IndexedRecord currentValue,
                                                                              org.apache.avro.Schema schema,
                                                                              Properties properties)
                                                                       throws IOException

Description copied from interface: HoodieRecordPayload

This methods lets you write custom merging/combining logic to produce new values as a function of current value on storage and whats contained in this object. Implementations can leverage properties if required.

eg: 1) You are updating counters, you may want to add counts to currentValue and write back updated counts 2) You may be reading DB redo logs, and merge them with current image for a database row on storage

Parameters:

currentValue - Current value in storage, to merge/combine this payload with

schema - Schema used for record

properties - Payload related properties. For example pass the ordering field(s) name to extract from value in storage.

Returns:

new combined/merged value to be written back to storage. EMPTY to skip writing this record.

Throws:

IOException

combineAndGetUpdateValue

public Option<org.apache.avro.generic.IndexedRecord> combineAndGetUpdateValue(org.apache.avro.generic.IndexedRecord currentValue,
                                                                              org.apache.avro.Schema schema)
                                                                       throws IOException

Description copied from interface: HoodieRecordPayload

This methods is deprecated. Please refer to HoodieRecordPayload.combineAndGetUpdateValue(IndexedRecord, Schema, Properties) for java docs.

Specified by:

combineAndGetUpdateValue in interface HoodieRecordPayload<OverwriteWithLatestAvroPayload>

Overrides:

combineAndGetUpdateValue in class OverwriteWithLatestAvroPayload

Throws:

IOException

isDeleteRecord

protected boolean isDeleteRecord(org.apache.avro.generic.GenericRecord record)

Overrides:

isDeleteRecord in class BaseAvroPayload

Parameters:

record - instance of GenericRecord of interest.