com.amazonaws.services.kinesis.connectors.s3

Class S3Emitter

java.lang.Object

com.amazonaws.services.kinesis.connectors.s3.S3Emitter

All Implemented Interfaces:

IEmitter<byte[]>

Direct Known Subclasses:

RedshiftBasicEmitter, S3ManifestEmitter

public class S3Emitter
extends Object
implements IEmitter<byte[]>

This implementation of IEmitter is used to store files from an Amazon Kinesis stream in S3. The use of this class requires the configuration of an Amazon S3 bucket/endpoint. When the buffer is full, this class's emit method adds the contents of the buffer to Amazon S3 as one file. The filename is generated from the first and last sequence numbers of the records contained in that file separated by a dash. This class requires the configuration of an Amazon S3 bucket and endpoint.

Field Summary

Fields

Modifier and Type	Field and Description
protected String	s3Bucket
protected com.amazonaws.services.s3.AmazonS3Client	s3client
protected String	s3Endpoint

Constructor Summary

Constructors

Constructor and Description
S3Emitter(KinesisConnectorConfiguration configuration)

Method Summary

Modifier and Type	Method and Description
List<byte[]>	emit(UnmodifiableBuffer<byte[]> buffer) Invoked when the buffer is full.
void	fail(List<byte[]> records) This method defines how to handle a set of records that cannot successfully be emitted.
protected String	getS3FileName(String firstSeq, String lastSeq)
protected String	getS3URI(String s3FileName)
void	shutdown() This method is called when the KinesisConnectorRecordProcessor is shutdown.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

s3Bucket

protected final String s3Bucket

s3Endpoint

protected final String s3Endpoint

s3client

protected final com.amazonaws.services.s3.AmazonS3Client s3client

Constructor Detail

S3Emitter

public S3Emitter(KinesisConnectorConfiguration configuration)

Method Detail

getS3FileName

protected String getS3FileName(String firstSeq,
                               String lastSeq)

getS3URI

protected String getS3URI(String s3FileName)

emit

public List<byte[]> emit(UnmodifiableBuffer<byte[]> buffer)
                  throws IOException

Description copied from interface: IEmitter

Invoked when the buffer is full. This method emits the set of filtered records. It should return a list of records that were not emitted successfully. Returning Collections.emptyList() is considered a success.

Specified by:

emit in interface IEmitter<byte[]>

Parameters:

buffer - The full buffer of records

Returns:

A list of records that failed to emit to be retried

Throws:

IOException - A failure was reached that is not recoverable, no retry will occur and the fail method will be called

fail

public void fail(List<byte[]> records)

Description copied from interface: IEmitter

This method defines how to handle a set of records that cannot successfully be emitted.

Specified by:

fail in interface IEmitter<byte[]>

Parameters:

records - a list of records that were not successfully emitted

shutdown

public void shutdown()

Description copied from interface: IEmitter

This method is called when the KinesisConnectorRecordProcessor is shutdown. It should close any existing client connections.

Specified by:

shutdown in interface IEmitter<byte[]>