software.amazon.kinesis.connectors.flink.internals.publisher.fanout

Class FanOutShardSubscriber

java.lang.Object

software.amazon.kinesis.connectors.flink.internals.publisher.fanout.FanOutShardSubscriber

@Internal
public class FanOutShardSubscriber
extends Object

This class is responsible for acquiring an Enhanced Fan Out subscription and consuming records from a shard. A queue is used to buffer records between the Kinesis Proxy and Flink application. This allows processing to be separated from consumption; errors thrown in the consumption layer do not propagate up to application.

 [
 | ----------- Source Connector Thread ----------- |                      | --- KinesisAsyncClient Thread(s) -- |
 | FanOutRecordPublisher | FanOutShardSubscription | == blocking queue == | KinesisProxyV2 | KinesisAsyncClient |
 ]

Three types of message are passed over the queue for inter-thread communication:

• SubscriptionNextEvent - passes data from the network to the consumer
• SubscriptionCompleteEvent - indicates a subscription has expired
• SubscriptionErrorEvent - passes an exception from the network to the consumer

The blocking queue has a maximum capacity of 1 record. This allows backpressure to be applied closer to the network stack and results in record prefetch. At maximum capacity we will have three SubscribeToShardEvent in memory (per instance of this class):

• 1 event being processed by the consumer
• 1 event enqueued in the blocking queue
• 1 event being added to the queue by the network (blocking)

Method Summary

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait