com.amazonaws.services.kinesis.connectors

Class KinesisConnectorRecordProcessor<T,U>

java.lang.Object

com.amazonaws.services.kinesis.connectors.KinesisConnectorRecordProcessor<T,U>

All Implemented Interfaces:

com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor

public class KinesisConnectorRecordProcessor<T,U>
extends Object
implements com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor

This is the base class for any KinesisConnector. It is configured by a constructor that takes in as parameters implementations of the IBuffer, ITransformer, and IEmitter dependencies defined in a IKinesisConnectorPipeline. It is typed to match the class that records are transformed into for filtering and manipulation. This class is produced by a KinesisConnectorRecordProcessorFactory.

When a Worker calls processRecords() on this class, the pipeline is used in the following way:

1. Records are transformed into the corresponding data model (parameter type T) via the ITransformer.
2. Transformed records are passed to the IBuffer.consumeRecord() method, which may optionally filter based on the IFilter in the pipeline.
3. When the buffer is full (IBuffer.shouldFlush() returns true), records are transformed with the ITransformer to the output type (parameter type U) and a call is made to IEmitter.emit(). IEmitter.emit() returning an empty list is considered a success, so the record processor will checkpoint and emit will not be retried. Non-empty return values will result in additional calls to emit with failed records as the unprocessed list until the retry limit is reached. Upon exceeding the retry limit or an exception being thrown, the IEmitter.fail() method will be called with the unprocessed records.
4. When the shutdown() method of this class is invoked, a call is made to the IEmitter.shutdown() method which should close any existing client connections.

Constructor Summary

Constructors

Constructor and Description
KinesisConnectorRecordProcessor(IBuffer<T> buffer, IFilter<T> filter, IEmitter<U> emitter, ITransformerBase<T,U> transformer, KinesisConnectorConfiguration configuration)

Method Summary

Modifier and Type	Method and Description
void	initialize(String shardId)
void	processRecords(List<com.amazonaws.services.kinesis.model.Record> records, com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorCheckpointer checkpointer)
void	shutdown(com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorCheckpointer checkpointer, com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShutdownReason reason)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

KinesisConnectorRecordProcessor

public KinesisConnectorRecordProcessor(IBuffer<T> buffer,
                                       IFilter<T> filter,
                                       IEmitter<U> emitter,
                                       ITransformerBase<T,U> transformer,
                                       KinesisConnectorConfiguration configuration)

Method Detail

initialize

public void initialize(String shardId)

Specified by:

initialize in interface com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor

processRecords

public void processRecords(List<com.amazonaws.services.kinesis.model.Record> records,
                           com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorCheckpointer checkpointer)

Specified by:

processRecords in interface com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor

shutdown

public void shutdown(com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorCheckpointer checkpointer,
                     com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShutdownReason reason)

Specified by:

shutdown in interface com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessor