| Modifier and Type | Method and Description |
|---|---|
static HoodieRecord |
HoodieAvroUtils.createHoodieRecordFromAvro(org.apache.avro.generic.IndexedRecord data,
String payloadClass,
String preCombineField,
Option<Pair<String,String>> simpleKeyGenFieldsOpt,
Boolean withOperation,
Option<String> partitionNameOp,
Boolean populateMetaFields,
Option<org.apache.avro.Schema> schemaWithoutMetaFields) |
| Modifier and Type | Method and Description |
|---|---|
static Option<byte[]> |
HoodieAvroUtils.recordToBytes(HoodieRecord record,
org.apache.avro.Schema schema)
TODO serialize other type of record.
|
| Modifier and Type | Class and Description |
|---|---|
class |
HoodieAvroIndexedRecord
This only use by reader returning.
|
class |
HoodieAvroRecord<T extends HoodieRecordPayload>
Implementation of
HoodieRecord using Avro payload. |
class |
HoodieEmptyRecord<T> |
| Modifier and Type | Method and Description |
|---|---|
HoodieRecord<T> |
HoodieEmptyRecord.copy() |
HoodieRecord<org.apache.avro.generic.IndexedRecord> |
HoodieAvroIndexedRecord.copy() |
abstract HoodieRecord<T> |
HoodieRecord.copy()
This is used to copy data.
|
HoodieRecord<T> |
HoodieAvroRecord.copy() |
HoodieRecord |
HoodieEmptyRecord.joinWith(HoodieRecord other,
org.apache.avro.Schema targetSchema) |
HoodieRecord |
HoodieAvroIndexedRecord.joinWith(HoodieRecord other,
org.apache.avro.Schema targetSchema) |
abstract HoodieRecord |
HoodieRecord.joinWith(HoodieRecord other,
org.apache.avro.Schema targetSchema)
Support bootstrap.
|
HoodieRecord |
HoodieAvroRecord.joinWith(HoodieRecord other,
org.apache.avro.Schema targetSchema) |
HoodieRecord<T> |
HoodieEmptyRecord.newInstance() |
HoodieRecord |
HoodieAvroIndexedRecord.newInstance() |
abstract HoodieRecord<T> |
HoodieRecord.newInstance() |
HoodieRecord<T> |
HoodieAvroRecord.newInstance() |
HoodieRecord<T> |
HoodieEmptyRecord.newInstance(HoodieKey key) |
HoodieRecord<org.apache.avro.generic.IndexedRecord> |
HoodieAvroIndexedRecord.newInstance(HoodieKey key) |
abstract HoodieRecord<T> |
HoodieRecord.newInstance(HoodieKey key) |
HoodieRecord<T> |
HoodieAvroRecord.newInstance(HoodieKey key) |
HoodieRecord<T> |
HoodieEmptyRecord.newInstance(HoodieKey key,
HoodieOperation op) |
HoodieRecord<org.apache.avro.generic.IndexedRecord> |
HoodieAvroIndexedRecord.newInstance(HoodieKey key,
HoodieOperation op) |
abstract HoodieRecord<T> |
HoodieRecord.newInstance(HoodieKey key,
HoodieOperation op) |
HoodieRecord<T> |
HoodieAvroRecord.newInstance(HoodieKey key,
HoodieOperation op) |
HoodieRecord |
HoodieEmptyRecord.prependMetaFields(org.apache.avro.Schema recordSchema,
org.apache.avro.Schema targetSchema,
MetadataValues metadataValues,
Properties props) |
HoodieRecord |
HoodieAvroIndexedRecord.prependMetaFields(org.apache.avro.Schema recordSchema,
org.apache.avro.Schema targetSchema,
MetadataValues metadataValues,
Properties props) |
abstract HoodieRecord |
HoodieRecord.prependMetaFields(org.apache.avro.Schema recordSchema,
org.apache.avro.Schema targetSchema,
MetadataValues metadataValues,
Properties props)
Rewrites record into new target schema containing Hudi-specific meta-fields
NOTE: This operation is idempotent
|
HoodieRecord |
HoodieAvroRecord.prependMetaFields(org.apache.avro.Schema recordSchema,
org.apache.avro.Schema targetSchema,
MetadataValues metadataValues,
Properties props) |
HoodieRecord |
HoodieRecord.rewriteRecordWithNewSchema(org.apache.avro.Schema recordSchema,
Properties props,
org.apache.avro.Schema newSchema) |
HoodieRecord |
HoodieEmptyRecord.rewriteRecordWithNewSchema(org.apache.avro.Schema recordSchema,
Properties props,
org.apache.avro.Schema newSchema,
Map<String,String> renameCols) |
HoodieRecord |
HoodieAvroIndexedRecord.rewriteRecordWithNewSchema(org.apache.avro.Schema recordSchema,
Properties props,
org.apache.avro.Schema newSchema,
Map<String,String> renameCols) |
abstract HoodieRecord |
HoodieRecord.rewriteRecordWithNewSchema(org.apache.avro.Schema recordSchema,
Properties props,
org.apache.avro.Schema newSchema,
Map<String,String> renameCols)
Support schema evolution.
|
HoodieRecord |
HoodieAvroRecord.rewriteRecordWithNewSchema(org.apache.avro.Schema recordSchema,
Properties props,
org.apache.avro.Schema newSchema,
Map<String,String> renameCols) |
HoodieRecord |
HoodieRecord.setCurrentLocation(HoodieRecordLocation location)
Sets the current currentLocation of the record.
|
HoodieRecord |
HoodieEmptyRecord.truncateRecordKey(org.apache.avro.Schema recordSchema,
Properties props,
String keyFieldName) |
HoodieRecord |
HoodieAvroIndexedRecord.truncateRecordKey(org.apache.avro.Schema recordSchema,
Properties props,
String keyFieldName) |
HoodieRecord |
HoodieAvroRecord.truncateRecordKey(org.apache.avro.Schema recordSchema,
Properties props,
String keyFieldName) |
HoodieRecord |
HoodieRecordCompatibilityInterface.truncateRecordKey(org.apache.avro.Schema recordSchema,
Properties props,
String keyFieldName)
This method used to overwrite record key field.
|
HoodieRecord |
HoodieEmptyRecord.wrapIntoHoodieRecordPayloadWithKeyGen(org.apache.avro.Schema recordSchema,
Properties props,
Option<BaseKeyGenerator> keyGen) |
HoodieRecord |
HoodieAvroIndexedRecord.wrapIntoHoodieRecordPayloadWithKeyGen(org.apache.avro.Schema recordSchema,
Properties props,
Option<BaseKeyGenerator> keyGen) |
HoodieRecord |
HoodieAvroRecord.wrapIntoHoodieRecordPayloadWithKeyGen(org.apache.avro.Schema recordSchema,
Properties props,
Option<BaseKeyGenerator> keyGen) |
HoodieRecord |
HoodieRecordCompatibilityInterface.wrapIntoHoodieRecordPayloadWithKeyGen(org.apache.avro.Schema recordSchema,
Properties props,
Option<BaseKeyGenerator> keyGen)
This method used to extract HoodieKey through keyGenerator.
|
HoodieRecord |
HoodieEmptyRecord.wrapIntoHoodieRecordPayloadWithParams(org.apache.avro.Schema recordSchema,
Properties props,
Option<Pair<String,String>> simpleKeyGenFieldsOpt,
Boolean withOperation,
Option<String> partitionNameOp,
Boolean populateMetaFieldsOp,
Option<org.apache.avro.Schema> schemaWithoutMetaFields) |
HoodieRecord |
HoodieAvroIndexedRecord.wrapIntoHoodieRecordPayloadWithParams(org.apache.avro.Schema recordSchema,
Properties props,
Option<Pair<String,String>> simpleKeyGenFieldsOpt,
Boolean withOperation,
Option<String> partitionNameOp,
Boolean populateMetaFields,
Option<org.apache.avro.Schema> schemaWithoutMetaFields) |
HoodieRecord |
HoodieAvroRecord.wrapIntoHoodieRecordPayloadWithParams(org.apache.avro.Schema recordSchema,
Properties props,
Option<Pair<String,String>> simpleKeyGenFieldsOpt,
Boolean withOperation,
Option<String> partitionNameOp,
Boolean populateMetaFields,
Option<org.apache.avro.Schema> schemaWithoutMetaFields) |
HoodieRecord |
HoodieRecordCompatibilityInterface.wrapIntoHoodieRecordPayloadWithParams(org.apache.avro.Schema recordSchema,
Properties props,
Option<Pair<String,String>> simpleKeyGenFieldsOpt,
Boolean withOperation,
Option<String> partitionNameOp,
Boolean populateMetaFieldsOp,
Option<org.apache.avro.Schema> schemaWithoutMetaFields)
This method used to extract HoodieKey not through keyGenerator.
|
| Modifier and Type | Method and Description |
|---|---|
Option<Pair<HoodieRecord,org.apache.avro.Schema>> |
HoodiePreCombineAvroRecordMerger.merge(HoodieRecord older,
org.apache.avro.Schema oldSchema,
HoodieRecord newer,
org.apache.avro.Schema newSchema,
TypedProperties props) |
Option<Pair<HoodieRecord,org.apache.avro.Schema>> |
HoodieAvroRecordMerger.merge(HoodieRecord older,
org.apache.avro.Schema oldSchema,
HoodieRecord newer,
org.apache.avro.Schema newSchema,
TypedProperties props) |
Option<Pair<HoodieRecord,org.apache.avro.Schema>> |
HoodieRecordMerger.merge(HoodieRecord older,
org.apache.avro.Schema oldSchema,
HoodieRecord newer,
org.apache.avro.Schema newSchema,
TypedProperties props)
This method converges combineAndGetUpdateValue and precombine from HoodiePayload.
|
| Modifier and Type | Method and Description |
|---|---|
static HoodieRecordDelegate |
HoodieRecordDelegate.fromHoodieRecord(HoodieRecord record) |
static HoodieRecordDelegate |
HoodieRecordDelegate.fromHoodieRecord(HoodieRecord record,
HoodieRecordLocation newLocationOverride) |
HoodieRecord |
HoodieEmptyRecord.joinWith(HoodieRecord other,
org.apache.avro.Schema targetSchema) |
HoodieRecord |
HoodieAvroIndexedRecord.joinWith(HoodieRecord other,
org.apache.avro.Schema targetSchema) |
abstract HoodieRecord |
HoodieRecord.joinWith(HoodieRecord other,
org.apache.avro.Schema targetSchema)
Support bootstrap.
|
HoodieRecord |
HoodieAvroRecord.joinWith(HoodieRecord other,
org.apache.avro.Schema targetSchema) |
Option<Pair<HoodieRecord,org.apache.avro.Schema>> |
HoodiePreCombineAvroRecordMerger.merge(HoodieRecord older,
org.apache.avro.Schema oldSchema,
HoodieRecord newer,
org.apache.avro.Schema newSchema,
TypedProperties props) |
Option<Pair<HoodieRecord,org.apache.avro.Schema>> |
HoodiePreCombineAvroRecordMerger.merge(HoodieRecord older,
org.apache.avro.Schema oldSchema,
HoodieRecord newer,
org.apache.avro.Schema newSchema,
TypedProperties props) |
Option<Pair<HoodieRecord,org.apache.avro.Schema>> |
HoodieAvroRecordMerger.merge(HoodieRecord older,
org.apache.avro.Schema oldSchema,
HoodieRecord newer,
org.apache.avro.Schema newSchema,
TypedProperties props) |
Option<Pair<HoodieRecord,org.apache.avro.Schema>> |
HoodieAvroRecordMerger.merge(HoodieRecord older,
org.apache.avro.Schema oldSchema,
HoodieRecord newer,
org.apache.avro.Schema newSchema,
TypedProperties props) |
Option<Pair<HoodieRecord,org.apache.avro.Schema>> |
HoodieRecordMerger.merge(HoodieRecord older,
org.apache.avro.Schema oldSchema,
HoodieRecord newer,
org.apache.avro.Schema newSchema,
TypedProperties props)
This method converges combineAndGetUpdateValue and precombine from HoodiePayload.
|
Option<Pair<HoodieRecord,org.apache.avro.Schema>> |
HoodieRecordMerger.merge(HoodieRecord older,
org.apache.avro.Schema oldSchema,
HoodieRecord newer,
org.apache.avro.Schema newSchema,
TypedProperties props)
This method converges combineAndGetUpdateValue and precombine from HoodiePayload.
|
| Constructor and Description |
|---|
HoodieAvroIndexedRecord(HoodieRecord<org.apache.avro.generic.IndexedRecord> record) |
HoodieAvroRecord(HoodieRecord<T> record) |
HoodieRecord(HoodieRecord<T> record) |
| Modifier and Type | Method and Description |
|---|---|
Map<String,HoodieRecord> |
HoodieMergedLogRecordScanner.getRecords() |
Iterator<HoodieRecord> |
HoodieMergedLogRecordScanner.iterator() |
| Modifier and Type | Method and Description |
|---|---|
void |
HoodieUnMergedLogRecordScanner.LogRecordScannerCallback.apply(HoodieRecord<?> record) |
abstract <T> void |
AbstractHoodieLogRecordReader.processNextRecord(HoodieRecord<T> hoodieRecord)
Process next record.
|
<T> void |
HoodieMergedLogRecordScanner.processNextRecord(HoodieRecord<T> newRecord) |
<T> void |
HoodieUnMergedLogRecordScanner.processNextRecord(HoodieRecord<T> hoodieRecord) |
| Modifier and Type | Method and Description |
|---|---|
protected <T> ClosableIterator<HoodieRecord<T>> |
HoodieParquetDataBlock.deserializeRecords(byte[] content,
HoodieRecord.HoodieRecordType type) |
protected abstract <T> ClosableIterator<HoodieRecord<T>> |
HoodieDataBlock.deserializeRecords(byte[] content,
HoodieRecord.HoodieRecordType type) |
protected <T> ClosableIterator<HoodieRecord<T>> |
HoodieHFileDataBlock.deserializeRecords(byte[] content,
HoodieRecord.HoodieRecordType type) |
protected <T> ClosableIterator<HoodieRecord<T>> |
HoodieAvroDataBlock.deserializeRecords(byte[] content,
HoodieRecord.HoodieRecordType type) |
<T> ClosableIterator<HoodieRecord<T>> |
HoodieDataBlock.getRecordIterator(HoodieRecord.HoodieRecordType type)
Returns all the records iterator contained w/in this block.
|
<T> ClosableIterator<HoodieRecord<T>> |
HoodieDataBlock.getRecordIterator(List<String> keys,
boolean fullKey,
HoodieRecord.HoodieRecordType type)
Batch get of keys of interest.
|
protected <T> ClosableIterator<HoodieRecord<T>> |
HoodieDataBlock.lookupRecords(List<String> keys,
boolean fullKey) |
protected <T> ClosableIterator<HoodieRecord<T>> |
HoodieHFileDataBlock.lookupRecords(List<String> sortedKeys,
boolean fullKey) |
protected <T> ClosableIterator<HoodieRecord<T>> |
HoodieParquetDataBlock.readRecordsFromBlockPayload(HoodieRecord.HoodieRecordType type)
NOTE: We're overriding the whole reading sequence to make sure we properly respect
the requested Reader's schema and only fetch the columns that have been explicitly
requested by the caller (providing projected Reader's schema)
|
protected <T> ClosableIterator<HoodieRecord<T>> |
HoodieDataBlock.readRecordsFromBlockPayload(HoodieRecord.HoodieRecordType type) |
| Modifier and Type | Method and Description |
|---|---|
protected Option<String> |
HoodieDataBlock.getRecordKey(HoodieRecord record) |
| Modifier and Type | Method and Description |
|---|---|
protected byte[] |
HoodieParquetDataBlock.serializeRecords(List<HoodieRecord> records,
HoodieStorage storage) |
protected abstract byte[] |
HoodieDataBlock.serializeRecords(List<HoodieRecord> records,
HoodieStorage storage) |
protected byte[] |
HoodieHFileDataBlock.serializeRecords(List<HoodieRecord> records,
HoodieStorage storage) |
protected byte[] |
HoodieAvroDataBlock.serializeRecords(List<HoodieRecord> records,
HoodieStorage storage) |
| Constructor and Description |
|---|
HoodieAvroDataBlock(List<HoodieRecord> records,
Map<HoodieLogBlock.HeaderMetadataType,String> header,
String keyField) |
HoodieAvroDataBlock(List<HoodieRecord> records,
org.apache.avro.Schema schema)
Deprecated.
|
HoodieCDCDataBlock(List<HoodieRecord> records,
Map<HoodieLogBlock.HeaderMetadataType,String> header,
String keyField) |
HoodieDataBlock(List<HoodieRecord> records,
Map<HoodieLogBlock.HeaderMetadataType,String> header,
Map<HoodieLogBlock.HeaderMetadataType,String> footer,
String keyFieldName)
NOTE: This ctor is used on the write-path (ie when records ought to be written into the log)
|
HoodieHFileDataBlock(List<HoodieRecord> records,
Map<HoodieLogBlock.HeaderMetadataType,String> header,
String compressionCodec,
StoragePath pathForReader,
boolean useNativeHFileReader) |
HoodieParquetDataBlock(List<HoodieRecord> records,
Map<HoodieLogBlock.HeaderMetadataType,String> header,
String keyField,
String compressionCodecName,
double expectedCompressionRatio,
boolean useDictionaryEncoding) |
| Modifier and Type | Method and Description |
|---|---|
static <R> HoodieRecord<R> |
SpillableMapUtils.convertToHoodieRecordPayload(org.apache.avro.generic.GenericRecord rec,
String payloadClazz,
String preCombineField,
boolean withOperationField)
Utility method to convert bytes to HoodieRecord using schema and payload class.
|
static <R> HoodieRecord<R> |
SpillableMapUtils.convertToHoodieRecordPayload(org.apache.avro.generic.GenericRecord record,
String payloadClazz,
String preCombineField,
boolean withOperationField,
Option<String> partitionName,
Option<org.apache.avro.Schema> schemaWithoutMetaFields) |
static <R> HoodieRecord<R> |
SpillableMapUtils.convertToHoodieRecordPayload(org.apache.avro.generic.GenericRecord record,
String payloadClazz,
String preCombineField,
Pair<String,String> recordKeyPartitionPathFieldPair,
boolean withOperationField,
Option<String> partitionName,
Option<org.apache.avro.Schema> schemaWithoutMetaFields)
Utility method to convert bytes to HoodieRecord using schema and payload class.
|
| Modifier and Type | Method and Description |
|---|---|
long |
HoodieRecordSizeEstimator.sizeEstimate(HoodieRecord<T> hoodieRecord) |
| Modifier and Type | Method and Description |
|---|---|
abstract byte[] |
FileFormatUtils.serializeRecordsToLogBlock(HoodieStorage storage,
List<HoodieRecord> records,
org.apache.avro.Schema writerSchema,
org.apache.avro.Schema readerSchema,
String keyFieldName,
Map<String,String> paramsMap)
Serializes Hudi records to the log block.
|
| Constructor and Description |
|---|
HoodieRecordMissingException(HoodieRecord record) |
| Modifier and Type | Field and Description |
|---|---|
protected ClosableIterator<HoodieRecord<T>> |
HoodieBootstrapRecordIterator.dataFileIterator |
protected ClosableIterator<HoodieRecord<T>> |
HoodieBootstrapRecordIterator.skeletonIterator |
| Modifier and Type | Method and Description |
|---|---|
HoodieRecord<T> |
HoodieBootstrapRecordIterator.next() |
| Modifier and Type | Method and Description |
|---|---|
default ClosableIterator<HoodieRecord<T>> |
HoodieFileReader.getRecordIterator() |
ClosableIterator<HoodieRecord<T>> |
HoodieBootstrapFileReader.getRecordIterator(org.apache.avro.Schema schema) |
default ClosableIterator<HoodieRecord<T>> |
HoodieFileReader.getRecordIterator(org.apache.avro.Schema readerSchema) |
ClosableIterator<HoodieRecord<T>> |
HoodieBootstrapFileReader.getRecordIterator(org.apache.avro.Schema readerSchema,
org.apache.avro.Schema requestedSchema) |
ClosableIterator<HoodieRecord<org.apache.avro.generic.IndexedRecord>> |
HoodieAvroFileReader.getRecordIterator(org.apache.avro.Schema readerSchema,
org.apache.avro.Schema requestedSchema) |
ClosableIterator<HoodieRecord<T>> |
HoodieFileReader.getRecordIterator(org.apache.avro.Schema readerSchema,
org.apache.avro.Schema requestedSchema) |
default ClosableIterator<HoodieRecord<T>> |
HoodieSeekingFileReader.getRecordsByKeyPrefixIterator(List<String> sortedKeyPrefixes) |
default ClosableIterator<HoodieRecord<T>> |
HoodieSeekingFileReader.getRecordsByKeyPrefixIterator(List<String> sortedKeyPrefixes,
org.apache.avro.Schema schema) |
ClosableIterator<HoodieRecord<org.apache.avro.generic.IndexedRecord>> |
HoodieNativeAvroHFileReader.getRecordsByKeyPrefixIterator(List<String> sortedKeyPrefixes,
org.apache.avro.Schema schema) |
default ClosableIterator<HoodieRecord<T>> |
HoodieSeekingFileReader.getRecordsByKeysIterator(List<String> sortedKeys) |
default ClosableIterator<HoodieRecord<T>> |
HoodieSeekingFileReader.getRecordsByKeysIterator(List<String> sortedKeys,
org.apache.avro.Schema schema) |
ClosableIterator<HoodieRecord<org.apache.avro.generic.IndexedRecord>> |
HoodieNativeAvroHFileReader.getRecordsByKeysIterator(List<String> sortedKeys,
org.apache.avro.Schema schema) |
| Modifier and Type | Method and Description |
|---|---|
default void |
HoodieFileWriter.write(String recordKey,
HoodieRecord record,
org.apache.avro.Schema schema) |
default void |
HoodieAvroFileWriter.write(String recordKey,
HoodieRecord record,
org.apache.avro.Schema schema,
Properties props) |
void |
HoodieFileWriter.write(String recordKey,
HoodieRecord record,
org.apache.avro.Schema schema,
Properties props) |
default void |
HoodieFileWriter.writeWithMetadata(HoodieKey key,
HoodieRecord record,
org.apache.avro.Schema schema) |
default void |
HoodieAvroFileWriter.writeWithMetadata(HoodieKey key,
HoodieRecord record,
org.apache.avro.Schema schema,
Properties props) |
void |
HoodieFileWriter.writeWithMetadata(HoodieKey key,
HoodieRecord record,
org.apache.avro.Schema schema,
Properties props) |
| Constructor and Description |
|---|
HoodieBootstrapRecordIterator(ClosableIterator<HoodieRecord<T>> skeletonIterator,
ClosableIterator<HoodieRecord<T>> dataFileIterator,
org.apache.avro.Schema schema,
Option<String[]> partitionFields,
Object[] partitionValues) |
HoodieBootstrapRecordIterator(ClosableIterator<HoodieRecord<T>> skeletonIterator,
ClosableIterator<HoodieRecord<T>> dataFileIterator,
org.apache.avro.Schema schema,
Option<String[]> partitionFields,
Object[] partitionValues) |
| Modifier and Type | Method and Description |
|---|---|
static HoodieRecord<HoodieMetadataPayload> |
HoodieMetadataPayload.createBloomFilterMetadataRecord(String partitionName,
String baseFileName,
String timestamp,
String bloomFilterType,
ByteBuffer bloomFilter,
boolean isDeleted)
Create bloom filter metadata record.
|
static HoodieRecord<HoodieMetadataPayload> |
HoodieMetadataPayload.createPartitionFilesRecord(String partition,
Map<String,Long> filesAdded,
List<String> filesDeleted)
Create and return a
HoodieMetadataPayload to save list of files within a partition. |
static HoodieRecord<HoodieMetadataPayload> |
HoodieMetadataPayload.createPartitionListRecord(List<String> partitions)
Create and return a
HoodieMetadataPayload to save list of partitions. |
static HoodieRecord<HoodieMetadataPayload> |
HoodieMetadataPayload.createPartitionListRecord(List<String> partitions,
boolean isDeleted)
Create and return a
HoodieMetadataPayload to save list of partitions. |
static HoodieRecord |
HoodieMetadataPayload.createRecordIndexDelete(String recordKey)
Create and return a
HoodieMetadataPayload to delete a record in the Metadata Table's record index. |
static HoodieRecord<HoodieMetadataPayload> |
HoodieMetadataPayload.createRecordIndexUpdate(String recordKey,
String partition,
String fileId,
String instantTime,
int fileIdEncoding)
Create and return a
HoodieMetadataPayload to insert or update an entry for the record index. |
| Modifier and Type | Method and Description |
|---|---|
static HoodieData<HoodieRecord> |
HoodieTableMetadataUtil.convertFilesToBloomFilterRecords(HoodieEngineContext engineContext,
Map<String,List<String>> partitionToDeletedFiles,
Map<String,Map<String,Long>> partitionToAppendedFiles,
String instantTime,
HoodieTableMetaClient dataMetaClient,
int bloomIndexParallelism,
String bloomFilterType)
Convert added and deleted files metadata to bloom filter index records.
|
static HoodieData<HoodieRecord> |
HoodieTableMetadataUtil.convertFilesToColumnStatsRecords(HoodieEngineContext engineContext,
Map<String,List<String>> partitionToDeletedFiles,
Map<String,Map<String,Long>> partitionToAppendedFiles,
HoodieTableMetaClient dataMetaClient,
boolean isColumnStatsIndexEnabled,
int columnStatsIndexParallelism,
List<String> targetColumnsForColumnStatsIndex)
Convert added and deleted action metadata to column stats index records.
|
protected static List<HoodieRecord> |
HoodieTableMetadataUtil.convertFilesToFilesPartitionRecords(Map<String,List<String>> partitionToDeletedFiles,
Map<String,Map<String,Long>> partitionToAppendedFiles,
String instantTime,
String operation)
Convert rollback action metadata to files partition records.
|
static HoodieData<HoodieRecord> |
HoodieTableMetadataUtil.convertMetadataToBloomFilterRecords(HoodieCleanMetadata cleanMetadata,
HoodieEngineContext engineContext,
String instantTime,
int bloomIndexParallelism)
Convert clean metadata to bloom filter index records.
|
static HoodieData<HoodieRecord> |
HoodieTableMetadataUtil.convertMetadataToBloomFilterRecords(HoodieEngineContext context,
HoodieConfig hoodieConfig,
HoodieCommitMetadata commitMetadata,
String instantTime,
HoodieTableMetaClient dataMetaClient,
String bloomFilterType,
int bloomIndexParallelism)
Convert commit action metadata to bloom filter records.
|
static HoodieData<HoodieRecord> |
HoodieTableMetadataUtil.convertMetadataToColumnStatsRecords(HoodieCleanMetadata cleanMetadata,
HoodieEngineContext engineContext,
HoodieTableMetaClient dataMetaClient,
boolean isColumnStatsIndexEnabled,
int columnStatsIndexParallelism,
List<String> targetColumnsForColumnStatsIndex)
Convert clean metadata to column stats index records.
|
static HoodieData<HoodieRecord> |
HoodieTableMetadataUtil.convertMetadataToColumnStatsRecords(HoodieCommitMetadata commitMetadata,
HoodieEngineContext engineContext,
HoodieTableMetaClient dataMetaClient,
boolean isColumnStatsIndexEnabled,
int columnStatsIndexParallelism,
List<String> targetColumnsForColumnStatsIndex) |
static List<HoodieRecord> |
HoodieTableMetadataUtil.convertMetadataToFilesPartitionRecords(HoodieCleanMetadata cleanMetadata,
String instantTime)
Finds all files that were deleted as part of a clean and creates metadata table records for them.
|
static List<HoodieRecord> |
HoodieTableMetadataUtil.convertMetadataToFilesPartitionRecords(HoodieCommitMetadata commitMetadata,
String instantTime)
Finds all new files/partitions created as part of commit and creates metadata table records for them.
|
static Map<MetadataPartitionType,HoodieData<HoodieRecord>> |
HoodieTableMetadataUtil.convertMetadataToRecords(HoodieEngineContext engineContext,
HoodieCleanMetadata cleanMetadata,
String instantTime,
HoodieTableMetaClient dataMetaClient,
List<MetadataPartitionType> enabledPartitionTypes,
int bloomIndexParallelism,
boolean isColumnStatsIndexEnabled,
int columnStatsIndexParallelism,
List<String> targetColumnsForColumnStatsIndex)
Convert the clean action to metadata records.
|
static Map<MetadataPartitionType,HoodieData<HoodieRecord>> |
HoodieTableMetadataUtil.convertMetadataToRecords(HoodieEngineContext context,
HoodieConfig hoodieConfig,
HoodieCommitMetadata commitMetadata,
String instantTime,
HoodieTableMetaClient dataMetaClient,
List<MetadataPartitionType> enabledPartitionTypes,
String bloomFilterType,
int bloomIndexParallelism,
boolean isColumnStatsIndexEnabled,
int columnStatsIndexParallelism,
List<String> targetColumnsForColumnStatsIndex)
Convert commit action to metadata records for the enabled partition types.
|
static Map<MetadataPartitionType,HoodieData<HoodieRecord>> |
HoodieTableMetadataUtil.convertMetadataToRecords(HoodieEngineContext engineContext,
HoodieTableMetaClient dataTableMetaClient,
HoodieRollbackMetadata rollbackMetadata,
String instantTime)
Convert rollback action metadata to metadata table records.
|
static Map<MetadataPartitionType,HoodieData<HoodieRecord>> |
HoodieTableMetadataUtil.convertMissingPartitionRecords(HoodieEngineContext engineContext,
List<String> deletedPartitions,
Map<String,Map<String,Long>> filesAdded,
Map<String,List<String>> filesDeleted,
String instantTime) |
static Stream<HoodieRecord> |
HoodieMetadataPayload.createColumnStatsRecords(String partitionName,
Collection<HoodieColumnRangeMetadata<Comparable>> columnRangeMetadataList,
boolean isDeleted) |
protected Option<HoodieRecord<HoodieMetadataPayload>> |
HoodieBackedTableMetadata.getRecordByKey(String key,
String partitionName) |
protected abstract Option<HoodieRecord<HoodieMetadataPayload>> |
BaseTableMetadata.getRecordByKey(String key,
String partitionName) |
List<HoodieRecord<HoodieMetadataPayload>> |
HoodieMetadataLogRecordReader.getRecords() |
Map<String,HoodieRecord<HoodieMetadataPayload>> |
HoodieMetadataLogRecordReader.getRecordsByKeyPrefixes(List<String> sortedKeyPrefixes) |
HoodieData<HoodieRecord<HoodieMetadataPayload>> |
FileSystemBackedTableMetadata.getRecordsByKeyPrefixes(List<String> keyPrefixes,
String partitionName,
boolean shouldLoadInMemory) |
HoodieData<HoodieRecord<HoodieMetadataPayload>> |
HoodieBackedTableMetadata.getRecordsByKeyPrefixes(List<String> keyPrefixes,
String partitionName,
boolean shouldLoadInMemory) |
HoodieData<HoodieRecord<HoodieMetadataPayload>> |
HoodieTableMetadata.getRecordsByKeyPrefixes(List<String> keyPrefixes,
String partitionName,
boolean shouldLoadInMemory)
Fetch records by key prefixes.
|
Map<String,HoodieRecord<HoodieMetadataPayload>> |
HoodieMetadataLogRecordReader.getRecordsByKeys(List<String> sortedKeys)
Fetches records identified by the provided list of keys in case these are present in
the delta-log blocks
|
protected Map<String,HoodieRecord<HoodieMetadataPayload>> |
HoodieBackedTableMetadata.getRecordsByKeys(List<String> keys,
String partitionName) |
protected abstract Map<String,HoodieRecord<HoodieMetadataPayload>> |
BaseTableMetadata.getRecordsByKeys(List<String> keys,
String partitionName) |
static HoodieData<HoodieRecord> |
HoodieTableMetadataUtil.readRecordKeysFromBaseFiles(HoodieEngineContext engineContext,
HoodieConfig config,
List<Pair<String,HoodieBaseFile>> partitionBaseFilePairs,
boolean forDelete,
int recordIndexMaxParallelism,
String basePath,
StorageConfiguration<?> configuration,
String activeModule)
Deprecated.
|
static HoodieData<HoodieRecord> |
HoodieTableMetadataUtil.readRecordKeysFromFileSlices(HoodieEngineContext engineContext,
List<Pair<String,FileSlice>> partitionFileSlicePairs,
boolean forDelete,
int recordIndexMaxParallelism,
String activeModule,
HoodieTableMetaClient metaClient,
EngineType engineType)
Reads the record keys from the given file slices and returns a
HoodieData of HoodieRecord to be updated in the metadata table. |
Copyright © 2024 The Apache Software Foundation. All rights reserved.