software.amazon.kinesis.connectors.flink.util

Class RecordEmitter<T extends org.apache.flink.streaming.runtime.operators.windowing.TimestampedValue>

java.lang.Object

software.amazon.kinesis.connectors.flink.util.RecordEmitter<T>

Type Parameters:

T -

All Implemented Interfaces:

Runnable

@Internal
public abstract class RecordEmitter<T extends org.apache.flink.streaming.runtime.operators.windowing.TimestampedValue>
extends Object
implements Runnable

Emitter that handles event time synchronization between producer threads.

Records are organized into per producer queues that will block when capacity is exhausted.

Records are emitted by selecting the oldest available element of all producer queues, as long as the timestamp does not exceed the current shared watermark plus allowed lookahead interval.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	RecordEmitter.RecordQueue<T> Accepts records from readers.

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_QUEUE_CAPACITY The default capacity of a single queue.

Constructor Summary

Constructors

Constructor and Description
RecordEmitter(int queueCapacity)

Method Summary

Modifier and Type	Method and Description
protected abstract void	emit(T record, RecordEmitter.RecordQueue<T> source) Emit the record.
protected void	emitRecords()
RecordEmitter.RecordQueue<T>	getQueue(int producerIndex) The queue for the given producer (i.e.
String	printInfo() Summary of emit queues that can be used for logging.
void	run() Run loop, does not return unless stop() was called.
void	setCurrentWatermark(long watermark) Set the current watermark.
void	setMaxLookaheadMillis(long maxLookaheadMillis) How far ahead of the watermark records should be emitted.
void	stop()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_QUEUE_CAPACITY

public static final int DEFAULT_QUEUE_CAPACITY

The default capacity of a single queue.

Larger queue size can lead to higher throughput, but also to very high heap space consumption, depending on the size of elements.

Note that this is difficult to tune, because it does not take into account the size of individual objects.

See Also:

Constant Field Values

Constructor Detail

RecordEmitter

public RecordEmitter(int queueCapacity)

Method Detail

getQueue

public RecordEmitter.RecordQueue<T> getQueue(int producerIndex)

The queue for the given producer (i.e. Kinesis shard consumer thread).

The producer may hold on to the queue for subsequent records.

Parameters:

producerIndex -

Returns:

setMaxLookaheadMillis

public void setMaxLookaheadMillis(long maxLookaheadMillis)

How far ahead of the watermark records should be emitted.

Needs to account for the latency of obtaining the global watermark.

Parameters:

maxLookaheadMillis -

setCurrentWatermark

public void setCurrentWatermark(long watermark)

Set the current watermark.

This watermark will be used to control which records to emit from the queues of pending elements. When an element timestamp is ahead of the watermark by at least maxLookaheadMillis, elements in that queue will wait until the watermark advances.

Parameters:

watermark -

run

public void run()

Run loop, does not return unless stop() was called.

Specified by:

run in interface Runnable

stop

public void stop()

emitRecords

protected void emitRecords()

emit

protected abstract void emit(T record,
                             RecordEmitter.RecordQueue<T> source)

Emit the record. This is specific to a connector, like the Kinesis data fetcher.

printInfo

public String printInfo()

Summary of emit queues that can be used for logging.