org.apache.hudi.common.table.log

Class HoodieMergedLogRecordScanner

java.lang.Object

org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader

org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner

All Implemented Interfaces:

Closeable, AutoCloseable, Iterable<HoodieRecord>

@NotThreadSafe
public class HoodieMergedLogRecordScanner
extends AbstractHoodieLogRecordReader
implements Iterable<HoodieRecord>, Closeable

Scans through all the blocks in a list of HoodieLogFile and builds up a compacted/merged list of records which will be used as a lookup table when merging the base columnar file with the redo log file.

NOTE: If readBlockLazily is turned on, does not merge, instead keeps reading log blocks and merges everything at once This is an optimization to avoid seek() back and forth to read new block (forward seek()) and lazily read content of seen block (reverse and forward seek()) during merge | | Read Block 1 Metadata | | Read Block 1 Data | | | Read Block 2 Metadata | | Read Block 2 Data | | I/O Pass 1 | ..................... | I/O Pass 2 | ................. | | | Read Block N Metadata | | Read Block N Data |

This results in two I/O passes over the log file.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HoodieMergedLogRecordScanner.Builder Builder used to build HoodieUnMergedLogRecordScanner.

Nested classes/interfaces inherited from class org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader

AbstractHoodieLogRecordReader.KeySpec

Field Summary

Fields

Modifier and Type	Field and Description
HoodieTimer	timer

Fields inherited from class org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader

forceFullScan, hoodieTableMetaClient, logFilePaths, preCombineField, readerSchema, recordMerger, recordType

Method Summary

Modifier and Type	Method and Description
void	close()
long	getNumMergedRecordsInLog()
Map<String,HoodieRecord>	getRecords()
HoodieRecord.HoodieRecordType	getRecordType()
long	getTotalTimeTakenToReadAndMergeBlocks()
Iterator<HoodieRecord>	iterator()
static HoodieMergedLogRecordScanner.Builder	newBuilder() Returns the builder for HoodieMergedLogRecordScanner.
protected void	processNextDeletedRecord(DeleteRecord deleteRecord) Process next deleted record.
<T> void	processNextRecord(HoodieRecord<T> newRecord) Process next record.
void	scan() Scans delta-log files processing blocks
void	scan(boolean skipProcessingBlocks)
void	scanByFullKeys(List<String> keys) Provides incremental scanning capability where only provided keys will be looked up in the delta-log files, scanned and subsequently materialized into the internal cache
void	scanByKeyPrefixes(List<String> keyPrefixes) Provides incremental scanning capability where only keys matching provided key-prefixes will be looked up in the delta-log files, scanned and subsequently materialized into the internal cache

Methods inherited from class org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader

getCurrentInstantLogBlocks, getPartitionNameOverride, getPayloadClassFQN, getPayloadProps, getProgress, getTotalCorruptBlocks, getTotalLogBlocks, getTotalLogFiles, getTotalLogRecords, getTotalRollbacks, getValidBlockInstants, isWithOperationField, scanInternal

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Field Detail

timer

public final HoodieTimer timer

Method Detail

scan

public final void scan()

Scans delta-log files processing blocks

scan

public final void scan(boolean skipProcessingBlocks)

scanByFullKeys

public void scanByFullKeys(List<String> keys)

Provides incremental scanning capability where only provided keys will be looked up in the delta-log files, scanned and subsequently materialized into the internal cache

Parameters:

keys - to be looked up

scanByKeyPrefixes

public void scanByKeyPrefixes(List<String> keyPrefixes)

Provides incremental scanning capability where only keys matching provided key-prefixes will be looked up in the delta-log files, scanned and subsequently materialized into the internal cache

Parameters:

keyPrefixes - to be looked up

iterator

public Iterator<HoodieRecord> iterator()

Specified by:

iterator in interface Iterable<HoodieRecord>

getRecords

public Map<String,HoodieRecord> getRecords()

getRecordType

public HoodieRecord.HoodieRecordType getRecordType()

getNumMergedRecordsInLog

public long getNumMergedRecordsInLog()

newBuilder

public static HoodieMergedLogRecordScanner.Builder newBuilder()

Returns the builder for HoodieMergedLogRecordScanner.

processNextRecord

public <T> void processNextRecord(HoodieRecord<T> newRecord)
                           throws IOException

Description copied from class: AbstractHoodieLogRecordReader

Process next record.

Specified by:

processNextRecord in class AbstractHoodieLogRecordReader

Parameters:

newRecord - Hoodie Record to process

Throws:

IOException

processNextDeletedRecord

protected void processNextDeletedRecord(DeleteRecord deleteRecord)

Description copied from class: AbstractHoodieLogRecordReader

Process next deleted record.

Specified by:

processNextDeletedRecord in class AbstractHoodieLogRecordReader

Parameters:

deleteRecord - Deleted record(hoodie key and ordering value)

getTotalTimeTakenToReadAndMergeBlocks

public long getTotalTimeTakenToReadAndMergeBlocks()

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable