com.amazonaws.services.kinesis.connectors.redshift

Class RedshiftManifestEmitter

java.lang.Object

com.amazonaws.services.kinesis.connectors.redshift.RedshiftManifestEmitter

All Implemented Interfaces:

IEmitter<String>

public class RedshiftManifestEmitter
extends Object
implements IEmitter<String>

This implementation of IEmitter collects filenames from an Amazon Kinesis stream that has been started by a S3ManifestEmitter. The RedshiftManifestEmitter processes the list of Amazon S3 file names, generates a manifest file and performs an Amazon Redshift copy. The Amazon Redshift copy is done using transactions to prevent duplication of objects in Amazon Redshift.
It follows the following procedure:

1. Write manifest file to Amazon S3
2. Begin Amazon Redshift transaction
3. If any files already exist in Amazon Redshift, return and checkpoint (this transaction already completed successfully so no need to duplicate)
4. Write file names to Amazon Redshift file table
5. Call Amazon Redshift copy
6. Commit Amazon Redshift Transaction

This class requires the configuration of an Amazon S3 bucket and endpoint, as well as the following Amazon Redshift items:

• Amazon Redshift URL
• username and password
• data table and key column (data table stores items from the manifest copy)
• file table and key column (file table is used to store file names to prevent duplicate entries)
• mandatory flag for Amazon Redshift copy
• the delimiter used for string parsing when inserting entries into Amazon Redshift

NOTE: Amazon S3 bucket and Amazon Redshift table must be in the same region for Manifest Copy.

Constructor Summary

Constructors

Constructor and Description
RedshiftManifestEmitter(KinesisConnectorConfiguration configuration)

Method Summary

Modifier and Type	Method and Description
List<String>	emit(UnmodifiableBuffer<String> buffer) Invoked when the buffer is full.
void	fail(List<String> records) This method defines how to handle a set of records that cannot successfully be emitted.
protected void	redshiftCopy(Connection conn, String manifestFile) Executes a, Amazon Redshift copy from Amazon S3 using a Manifest file with a command in the format: COPY dataTable FROM 's3://s3Bucket/manifestFile' CREDENTIALS 'aws_access_key_id=accessKey;aws_secret_access_key=secretKey' DELIMITER dataDelimiter MANIFEST;
void	shutdown() This method is called when the KinesisConnectorRecordProcessor is shutdown.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RedshiftManifestEmitter

public RedshiftManifestEmitter(KinesisConnectorConfiguration configuration)

Method Detail

emit

public List<String> emit(UnmodifiableBuffer<String> buffer)
                  throws IOException

Description copied from interface: IEmitter

Invoked when the buffer is full. This method emits the set of filtered records. It should return a list of records that were not emitted successfully. Returning Collections.emptyList() is considered a success.

Specified by:

emit in interface IEmitter<String>

Parameters:

buffer - The full buffer of records

Returns:

A list of records that failed to emit to be retried

Throws:

IOException - A failure was reached that is not recoverable, no retry will occur and the fail method will be called

fail

public void fail(List<String> records)

Description copied from interface: IEmitter

This method defines how to handle a set of records that cannot successfully be emitted.

Specified by:

fail in interface IEmitter<String>

Parameters:

records - a list of records that were not successfully emitted

redshiftCopy

protected void redshiftCopy(Connection conn,
                            String manifestFile)
                     throws SQLException

Executes a, Amazon Redshift copy from Amazon S3 using a Manifest file with a command in the format: COPY dataTable FROM 's3://s3Bucket/manifestFile' CREDENTIALS 'aws_access_key_id=accessKey;aws_secret_access_key=secretKey' DELIMITER dataDelimiter MANIFEST;

Parameters:

Name - of manifest file

Throws:

SQLException

shutdown

public void shutdown()

Description copied from interface: IEmitter

This method is called when the KinesisConnectorRecordProcessor is shutdown. It should close any existing client connections.

Specified by:

shutdown in interface IEmitter<String>