com.amazonaws.services.kinesis.connectors.elasticsearch

Class ElasticsearchEmitter

java.lang.Object

com.amazonaws.services.kinesis.connectors.elasticsearch.ElasticsearchEmitter

All Implemented Interfaces:

IEmitter<ElasticsearchObject>

public class ElasticsearchEmitter
extends Object
implements IEmitter<ElasticsearchObject>

Constructor Summary

Constructors

Constructor and Description
ElasticsearchEmitter(KinesisConnectorConfiguration configuration)

Method Summary

Modifier and Type	Method and Description
List<ElasticsearchObject>	emit(UnmodifiableBuffer<ElasticsearchObject> buffer) Emits records to elasticsearch.
void	fail(List<ElasticsearchObject> records) This method defines how to handle a set of records that cannot successfully be emitted.
void	shutdown() This method is called when the KinesisConnectorRecordProcessor is shutdown.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ElasticsearchEmitter

public ElasticsearchEmitter(KinesisConnectorConfiguration configuration)

Method Detail

emit

public List<ElasticsearchObject> emit(UnmodifiableBuffer<ElasticsearchObject> buffer)
                               throws IOException

Emits records to elasticsearch. 1. Adds each record to a bulk index request, conditionally adding version, ttl or create if they were set in the transformer. 2. Executes the bulk request, returning any record specific failures to be retried by the connector library pipeline, unless outlined below. Record specific failures (noted in the failure.getMessage() string) - DocumentAlreadyExistsException means the record has create set to true, but a document already existed at the specific index/type/id. - VersionConflictEngineException means the record has a specific version number that did not match what existed in elasticsearch. To guarantee in order processing by the connector, when putting data use the same partition key for objects going to the same index/type/id and set sequence number for ordering. - In either case, the emitter will assume that the record would fail again in the future and thus will not return the record to be retried. Bulk request failures - NoNodeAvailableException means the TransportClient could not connect to the cluster. - A general Exception catches any other unexpected behavior. - In either case the emitter will continue making attempts until the issue has been resolved. This is to ensure that no data loss occurs and simplifies restarting the application once issues have been fixed.

Specified by:

emit in interface IEmitter<ElasticsearchObject>

Parameters:

buffer - The full buffer of records

Returns:

A list of records that failed to emit to be retried

Throws:

IOException - A failure was reached that is not recoverable, no retry will occur and the fail method will be called

fail

public void fail(List<ElasticsearchObject> records)

Description copied from interface: IEmitter

This method defines how to handle a set of records that cannot successfully be emitted.

Specified by:

fail in interface IEmitter<ElasticsearchObject>

Parameters:

records - a list of records that were not successfully emitted

shutdown

public void shutdown()

Description copied from interface: IEmitter

This method is called when the KinesisConnectorRecordProcessor is shutdown. It should close any existing client connections.

Specified by:

shutdown in interface IEmitter<ElasticsearchObject>