org.apache.hudi.common.table.log.block

Class HoodieParquetDataBlock

java.lang.Object

org.apache.hudi.common.table.log.block.HoodieLogBlock

org.apache.hudi.common.table.log.block.HoodieDataBlock

org.apache.hudi.common.table.log.block.HoodieParquetDataBlock

public class HoodieParquetDataBlock
extends HoodieDataBlock

HoodieParquetDataBlock contains a list of records serialized using Parquet.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hudi.common.table.log.block.HoodieLogBlock

HoodieLogBlock.FooterMetadataType, HoodieLogBlock.HeaderMetadataType, HoodieLogBlock.HoodieLogBlockContentLocation, HoodieLogBlock.HoodieLogBlockType

Field Summary

Fields inherited from class org.apache.hudi.common.table.log.block.HoodieDataBlock

readerSchema

Fields inherited from class org.apache.hudi.common.table.log.block.HoodieLogBlock

readBlockLazily, version

Constructor Summary

Constructors

Constructor and Description
HoodieParquetDataBlock(List<HoodieRecord> records, Map<HoodieLogBlock.HeaderMetadataType,String> header, String keyField, String compressionCodecName, double expectedCompressionRatio, boolean useDictionaryEncoding)
HoodieParquetDataBlock(Supplier<SeekableDataInputStream> inputStreamSupplier, Option<byte[]> content, boolean readBlockLazily, HoodieLogBlock.HoodieLogBlockContentLocation logBlockContentLocation, Option<org.apache.avro.Schema> readerSchema, Map<HoodieLogBlock.HeaderMetadataType,String> header, Map<HoodieLogBlock.HeaderMetadataType,String> footer, String keyField)

Method Summary

Modifier and Type	Method and Description
protected <T> ClosableIterator<HoodieRecord<T>>	deserializeRecords(byte[] content, HoodieRecord.HoodieRecordType type)
HoodieLogBlock.HoodieLogBlockType	getBlockType()
protected <T> ClosableIterator<HoodieRecord<T>>	readRecordsFromBlockPayload(HoodieRecord.HoodieRecordType type) NOTE: We're overriding the whole reading sequence to make sure we properly respect the requested Reader's schema and only fetch the columns that have been explicitly requested by the caller (providing projected Reader's schema)
protected byte[]	serializeRecords(List<HoodieRecord> records, HoodieStorage storage)

Methods inherited from class org.apache.hudi.common.table.log.block.HoodieDataBlock

getContentBytes, getKeyField, getKeyFieldName, getRecordIterator, getRecordIterator, getRecordKey, getSchema, getSchemaFromHeader, getWriterSchema, lookupRecords

Methods inherited from class org.apache.hudi.common.table.log.block.HoodieLogBlock

deflate, getBlockContentLocation, getContent, getLogBlockFooter, getLogBlockHeader, getLogBlockLength, getLogMetadata, getLogMetadataBytes, getMagic, getRecordPositions, inflate, isCompactedLogBlock, tryReadContent

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HoodieParquetDataBlock

public HoodieParquetDataBlock(Supplier<SeekableDataInputStream> inputStreamSupplier,
                              Option<byte[]> content,
                              boolean readBlockLazily,
                              HoodieLogBlock.HoodieLogBlockContentLocation logBlockContentLocation,
                              Option<org.apache.avro.Schema> readerSchema,
                              Map<HoodieLogBlock.HeaderMetadataType,String> header,
                              Map<HoodieLogBlock.HeaderMetadataType,String> footer,
                              String keyField)

HoodieParquetDataBlock

public HoodieParquetDataBlock(List<HoodieRecord> records,
                              Map<HoodieLogBlock.HeaderMetadataType,String> header,
                              String keyField,
                              String compressionCodecName,
                              double expectedCompressionRatio,
                              boolean useDictionaryEncoding)

Method Detail

getBlockType

public HoodieLogBlock.HoodieLogBlockType getBlockType()

Specified by:

getBlockType in class HoodieDataBlock

serializeRecords

protected byte[] serializeRecords(List<HoodieRecord> records,
                                  HoodieStorage storage)
                           throws IOException

Specified by:

serializeRecords in class HoodieDataBlock

Throws:

IOException

readRecordsFromBlockPayload

protected <T> ClosableIterator<HoodieRecord<T>> readRecordsFromBlockPayload(HoodieRecord.HoodieRecordType type)
                                                                     throws IOException

NOTE: We're overriding the whole reading sequence to make sure we properly respect the requested Reader's schema and only fetch the columns that have been explicitly requested by the caller (providing projected Reader's schema)

Overrides:

readRecordsFromBlockPayload in class HoodieDataBlock

Throws:

IOException

deserializeRecords

protected <T> ClosableIterator<HoodieRecord<T>> deserializeRecords(byte[] content,
                                                                   HoodieRecord.HoodieRecordType type)
                                                            throws IOException

Specified by:

deserializeRecords in class HoodieDataBlock

Throws:

IOException