org.apache.hudi.common.table.log.block

Class HoodieDataBlock

java.lang.Object

org.apache.hudi.common.table.log.block.HoodieLogBlock

org.apache.hudi.common.table.log.block.HoodieDataBlock

Direct Known Subclasses:

HoodieAvroDataBlock, HoodieHFileDataBlock, HoodieParquetDataBlock

public abstract class HoodieDataBlock
extends HoodieLogBlock

DataBlock contains a list of records serialized using formats compatible with the base file format. For each base file format there is a corresponding DataBlock format.

The Datablock contains: 1. Data Block version 2. Total number of records in the block 3. Actual serialized content of the records

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hudi.common.table.log.block.HoodieLogBlock

HoodieLogBlock.FooterMetadataType, HoodieLogBlock.HeaderMetadataType, HoodieLogBlock.HoodieLogBlockContentLocation, HoodieLogBlock.HoodieLogBlockType

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.avro.Schema	readerSchema

Fields inherited from class org.apache.hudi.common.table.log.block.HoodieLogBlock

readBlockLazily, version

Constructor Summary

Constructors

Modifier	Constructor and Description
	HoodieDataBlock(List<HoodieRecord> records, Map<HoodieLogBlock.HeaderMetadataType,String> header, Map<HoodieLogBlock.HeaderMetadataType,String> footer, String keyFieldName) NOTE: This ctor is used on the write-path (ie when records ought to be written into the log)
protected	HoodieDataBlock(Option<byte[]> content, Supplier<SeekableDataInputStream> inputStreamSupplier, boolean readBlockLazily, Option<HoodieLogBlock.HoodieLogBlockContentLocation> blockContentLocation, Option<org.apache.avro.Schema> readerSchema, Map<HoodieLogBlock.HeaderMetadataType,String> headers, Map<HoodieLogBlock.HeaderMetadataType,String> footer, String keyFieldName, boolean enablePointLookups) NOTE: This ctor is used on the write-path (ie when records ought to be written into the log)

Method Summary

Modifier and Type	Method and Description
protected abstract <T> ClosableIterator<HoodieRecord<T>>	deserializeRecords(byte[] content, HoodieRecord.HoodieRecordType type)
abstract HoodieLogBlock.HoodieLogBlockType	getBlockType()
byte[]	getContentBytes(HoodieStorage storage)
protected Option<org.apache.avro.Schema.Field>	getKeyField(org.apache.avro.Schema schema)
String	getKeyFieldName()
<T> ClosableIterator<HoodieRecord<T>>	getRecordIterator(HoodieRecord.HoodieRecordType type) Returns all the records iterator contained w/in this block.
<T> ClosableIterator<HoodieRecord<T>>	getRecordIterator(List<String> keys, boolean fullKey, HoodieRecord.HoodieRecordType type) Batch get of keys of interest.
protected Option<String>	getRecordKey(HoodieRecord record)
org.apache.avro.Schema	getSchema()
protected org.apache.avro.Schema	getSchemaFromHeader()
protected static org.apache.avro.Schema	getWriterSchema(Map<HoodieLogBlock.HeaderMetadataType,String> logBlockHeader)
protected <T> ClosableIterator<HoodieRecord<T>>	lookupRecords(List<String> keys, boolean fullKey)
protected <T> ClosableIterator<HoodieRecord<T>>	readRecordsFromBlockPayload(HoodieRecord.HoodieRecordType type)
protected abstract byte[]	serializeRecords(List<HoodieRecord> records, HoodieStorage storage)

Methods inherited from class org.apache.hudi.common.table.log.block.HoodieLogBlock

deflate, getBlockContentLocation, getContent, getLogBlockFooter, getLogBlockHeader, getLogBlockLength, getLogMetadata, getLogMetadataBytes, getMagic, getRecordPositions, inflate, isCompactedLogBlock, tryReadContent

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

readerSchema

protected org.apache.avro.Schema readerSchema

Constructor Detail

HoodieDataBlock

public HoodieDataBlock(List<HoodieRecord> records,
                       Map<HoodieLogBlock.HeaderMetadataType,String> header,
                       Map<HoodieLogBlock.HeaderMetadataType,String> footer,
                       String keyFieldName)

NOTE: This ctor is used on the write-path (ie when records ought to be written into the log)

HoodieDataBlock

protected HoodieDataBlock(Option<byte[]> content,
                          Supplier<SeekableDataInputStream> inputStreamSupplier,
                          boolean readBlockLazily,
                          Option<HoodieLogBlock.HoodieLogBlockContentLocation> blockContentLocation,
                          Option<org.apache.avro.Schema> readerSchema,
                          Map<HoodieLogBlock.HeaderMetadataType,String> headers,
                          Map<HoodieLogBlock.HeaderMetadataType,String> footer,
                          String keyFieldName,
                          boolean enablePointLookups)

NOTE: This ctor is used on the write-path (ie when records ought to be written into the log)

Method Detail

getContentBytes

public byte[] getContentBytes(HoodieStorage storage)
                       throws IOException

Overrides:

getContentBytes in class HoodieLogBlock

Throws:

IOException

getKeyFieldName

public String getKeyFieldName()

getWriterSchema

protected static org.apache.avro.Schema getWriterSchema(Map<HoodieLogBlock.HeaderMetadataType,String> logBlockHeader)

getRecordIterator

public final <T> ClosableIterator<HoodieRecord<T>> getRecordIterator(HoodieRecord.HoodieRecordType type)

Returns all the records iterator contained w/in this block.

getSchema

public org.apache.avro.Schema getSchema()

getRecordIterator

public final <T> ClosableIterator<HoodieRecord<T>> getRecordIterator(List<String> keys,
                                                                     boolean fullKey,
                                                                     HoodieRecord.HoodieRecordType type)
                                                              throws IOException

Batch get of keys of interest. Implementation can choose to either do full scan and return matched entries or do a seek based parsing and return matched entries.

Parameters:

keys - keys of interest.

Returns:

List of IndexedRecords for the keys of interest.

Throws:

IOException - in case of failures encountered when reading/parsing records

readRecordsFromBlockPayload

protected <T> ClosableIterator<HoodieRecord<T>> readRecordsFromBlockPayload(HoodieRecord.HoodieRecordType type)
                                                                     throws IOException

Throws:

IOException

lookupRecords

protected <T> ClosableIterator<HoodieRecord<T>> lookupRecords(List<String> keys,
                                                              boolean fullKey)
                                                       throws IOException

Throws:

IOException

serializeRecords

protected abstract byte[] serializeRecords(List<HoodieRecord> records,
                                           HoodieStorage storage)
                                    throws IOException

Throws:

IOException

deserializeRecords

protected abstract <T> ClosableIterator<HoodieRecord<T>> deserializeRecords(byte[] content,
                                                                            HoodieRecord.HoodieRecordType type)
                                                                     throws IOException

Throws:

IOException

getBlockType

public abstract HoodieLogBlock.HoodieLogBlockType getBlockType()

Specified by:

getBlockType in class HoodieLogBlock

getKeyField

protected Option<org.apache.avro.Schema.Field> getKeyField(org.apache.avro.Schema schema)

getRecordKey

protected Option<String> getRecordKey(HoodieRecord record)

getSchemaFromHeader

protected org.apache.avro.Schema getSchemaFromHeader()