org.apache.hudi.common.util.queue

Class BoundedInMemoryQueue<I,O>

java.lang.Object

org.apache.hudi.common.util.queue.BoundedInMemoryQueue<I,O>

Type Parameters:

I - input payload data type

O - output payload data type

All Implemented Interfaces:

Closeable, AutoCloseable, Iterable<O>, HoodieMessageQueue<I,O>

public class BoundedInMemoryQueue<I,O>
extends Object
implements HoodieMessageQueue<I,O>, Iterable<O>

Used for enqueueing input records. Queue limit is controlled by memoryLimit. Unlike standard bounded queue implementations, this queue bounds the size by memory bytes occupied by its tenants. The standard implementation bounds by the number of entries in the queue. It internally samples every RECORD_SAMPLING_RATEth record and adjusts number of records in queue accordingly. This is done to ensure that we don't OOM. This queue supports multiple producer single consumer pattern.

Field Summary

Fields

Modifier and Type	Field and Description
long	avgRecordSizeInBytes Indicates avg record size in bytes.
int	currentRateLimit indicates rate limit (number of records to cache).
Semaphore	rateLimiter It indicates number of records to cache.
static int	RECORD_POLL_INTERVAL_SEC Interval used for polling records in the queue.
static int	RECORD_SAMPLING_RATE Rate used for sampling records to determine avg record size in bytes.
AtomicLong	samplingRecordCounter Used for sampling records with "recordSamplingRate" frequency.

Constructor Summary

Constructors

Constructor and Description
BoundedInMemoryQueue(long memoryLimit, Function<I,O> transformFunction) Construct BoundedInMemoryQueue with default SizeEstimator.
BoundedInMemoryQueue(long memoryLimit, Function<I,O> transformFunction, SizeEstimator<O> payloadSizeEstimator)
BoundedInMemoryQueue(long memoryLimit, Function<I,O> transformFunction, SizeEstimator<O> payloadSizeEstimator, int recordSamplingRate, int recordCacheLimit) Construct BoundedInMemoryQueue with passed in size estimator.

Method Summary

Modifier and Type	Method and Description
void	close() Shuts down the queue, cleaning up the resources
Throwable	getThrowable()
void	insertRecord(I t) Inserts record into queue after applying transformation.
boolean	isEmpty()
Iterator<O>	iterator()
void	markAsFailed(Throwable e) API to allow producers and consumer to communicate termination due to failure.
Option<O>	readNextRecord() Reader interface but never exposed to outside world as this is a single consumer queue.
void	seal() Puts an empty entry to queue to denote termination.
long	size() Returns the number of elements in this queue.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Field Detail

RECORD_POLL_INTERVAL_SEC

public static final int RECORD_POLL_INTERVAL_SEC

Interval used for polling records in the queue.

See Also:

Constant Field Values

RECORD_SAMPLING_RATE

public static final int RECORD_SAMPLING_RATE

Rate used for sampling records to determine avg record size in bytes.

See Also:

Constant Field Values

rateLimiter

public final Semaphore rateLimiter

It indicates number of records to cache. We will be using sampled record's average size to determine how many records we should cache and will change (increase/decrease) permits accordingly.

samplingRecordCounter

public final AtomicLong samplingRecordCounter

Used for sampling records with "recordSamplingRate" frequency.

currentRateLimit

public int currentRateLimit

indicates rate limit (number of records to cache). it is updated whenever there is a change in avg record size.

avgRecordSizeInBytes

public long avgRecordSizeInBytes

Indicates avg record size in bytes. It is updated whenever a new record is sampled.

Constructor Detail

BoundedInMemoryQueue

public BoundedInMemoryQueue(long memoryLimit,
                            Function<I,O> transformFunction)

Construct BoundedInMemoryQueue with default SizeEstimator.

Parameters:

memoryLimit - MemoryLimit in bytes

transformFunction - Transformer Function to convert input payload type to stored payload type

BoundedInMemoryQueue

public BoundedInMemoryQueue(long memoryLimit,
                            Function<I,O> transformFunction,
                            SizeEstimator<O> payloadSizeEstimator)

BoundedInMemoryQueue

public BoundedInMemoryQueue(long memoryLimit,
                            Function<I,O> transformFunction,
                            SizeEstimator<O> payloadSizeEstimator,
                            int recordSamplingRate,
                            int recordCacheLimit)

Construct BoundedInMemoryQueue with passed in size estimator.

Parameters:

memoryLimit - MemoryLimit in bytes

transformFunction - Transformer Function to convert input payload type to stored payload type

payloadSizeEstimator - Payload Size Estimator

recordSamplingRate - record sampling rate

recordCacheLimit - record cache limit

Method Detail

size

public long size()

Description copied from interface: HoodieMessageQueue

Returns the number of elements in this queue.

Specified by:

size in interface HoodieMessageQueue<I,O>

insertRecord

public void insertRecord(I t)
                  throws Exception

Inserts record into queue after applying transformation.

Specified by:

insertRecord in interface HoodieMessageQueue<I,O>

Parameters:

t - Item to be queued

Throws:

Exception

readNextRecord

public Option<O> readNextRecord()

Reader interface but never exposed to outside world as this is a single consumer queue. Reading is done through a singleton iterator for this queue.

Specified by:

readNextRecord in interface HoodieMessageQueue<I,O>

seal

public void seal()

Puts an empty entry to queue to denote termination.

Specified by:

seal in interface HoodieMessageQueue<I,O>

close

public void close()

Description copied from interface: HoodieMessageQueue

Shuts down the queue, cleaning up the resources

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Specified by:

close in interface HoodieMessageQueue<I,O>

markAsFailed

public void markAsFailed(Throwable e)

API to allow producers and consumer to communicate termination due to failure.

Specified by:

markAsFailed in interface HoodieMessageQueue<I,O>

getThrowable

public Throwable getThrowable()

Specified by:

getThrowable in interface HoodieMessageQueue<I,O>

isEmpty

public boolean isEmpty()

Specified by:

isEmpty in interface HoodieMessageQueue<I,O>

iterator

public Iterator<O> iterator()

Specified by:

iterator in interface Iterable<O>