com.amazonaws.services.kinesis.connectors.s3

Class S3ManifestEmitter

java.lang.Object

com.amazonaws.services.kinesis.connectors.s3.S3Emitter

com.amazonaws.services.kinesis.connectors.s3.S3ManifestEmitter

All Implemented Interfaces:

IEmitter<byte[]>

public class S3ManifestEmitter
extends S3Emitter

This implementaion of IEmitter inserts records into Amazon S3 and emits filenames into a separate Amazon Kinesis stream. The separate Amazon Kinesis stream is to be used by another Amazon Kinesis enabled application that utilizes RedshiftManifestEmitters to insert the records into Amazon Redshift via a manifest copy. This class requires the configuration of an Amazon S3 bucket and endpoint, as well as Amazon Kinesis endpoint and output stream.

When the buffer is full, this Emitter:

1. Puts all records into a single file in S3
2. Puts the single file name into the manifest stream

NOTE: the Amazon S3 bucket and Amazon Redshift cluster must be in the same region.

Field Summary

Fields inherited from class com.amazonaws.services.kinesis.connectors.s3.S3Emitter

s3Bucket, s3client, s3Endpoint

Constructor Summary

Constructors

Constructor and Description
S3ManifestEmitter(KinesisConnectorConfiguration configuration)

Method Summary

Modifier and Type	Method and Description
List<byte[]>	emit(UnmodifiableBuffer<byte[]> buffer) Invoked when the buffer is full.
void	fail(List<byte[]> records) This method defines how to handle a set of records that cannot successfully be emitted.
void	shutdown() This method is called when the KinesisConnectorRecordProcessor is shutdown.

Methods inherited from class com.amazonaws.services.kinesis.connectors.s3.S3Emitter

getS3FileName, getS3URI

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

S3ManifestEmitter

public S3ManifestEmitter(KinesisConnectorConfiguration configuration)

Method Detail

emit

public List<byte[]> emit(UnmodifiableBuffer<byte[]> buffer)
                  throws IOException

Description copied from interface: IEmitter

Invoked when the buffer is full. This method emits the set of filtered records. It should return a list of records that were not emitted successfully. Returning Collections.emptyList() is considered a success.

Specified by:

emit in interface IEmitter<byte[]>

Overrides:

emit in class S3Emitter

Parameters:

buffer - The full buffer of records

Returns:

A list of records that failed to emit to be retried

Throws:

IOException - A failure was reached that is not recoverable, no retry will occur and the fail method will be called

fail

public void fail(List<byte[]> records)

Description copied from interface: IEmitter

This method defines how to handle a set of records that cannot successfully be emitted.

Specified by:

fail in interface IEmitter<byte[]>

Overrides:

fail in class S3Emitter

Parameters:

records - a list of records that were not successfully emitted

shutdown

public void shutdown()

Description copied from interface: IEmitter

This method is called when the KinesisConnectorRecordProcessor is shutdown. It should close any existing client connections.

Specified by:

shutdown in interface IEmitter<byte[]>

Overrides:

shutdown in class S3Emitter