software.amazon.kinesis.connectors.flink

Class FlinkKinesisProducer<OUT>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.sink.RichSinkFunction<OUT>

software.amazon.kinesis.connectors.flink.FlinkKinesisProducer<OUT>

Type Parameters:

OUT - Data type to produce into Kinesis Streams

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.functions.sink.SinkFunction<OUT>

@PublicEvolving
public class FlinkKinesisProducer<OUT>
extends org.apache.flink.streaming.api.functions.sink.RichSinkFunction<OUT>
implements org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

The FlinkKinesisProducer allows to produce from a Flink DataStream into Kinesis.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.sink.SinkFunction

org.apache.flink.streaming.api.functions.sink.SinkFunction.Context<T>

Field Summary

Fields

Modifier and Type	Field and Description
static String	KINESIS_PRODUCER_METRIC_GROUP
static String	METRIC_BACKPRESSURE_CYCLES
static String	METRIC_OUTSTANDING_RECORDS_COUNT

Constructor Summary

Constructors

Constructor and Description
FlinkKinesisProducer(KinesisSerializationSchema<OUT> schema, Properties configProps) Create a new FlinkKinesisProducer.
FlinkKinesisProducer(org.apache.flink.api.common.serialization.SerializationSchema<OUT> schema, Properties configProps) Create a new FlinkKinesisProducer.

Method Summary

Modifier and Type	Method and Description
void	close()
protected com.amazonaws.services.kinesis.producer.KinesisProducer	getKinesisProducer(com.amazonaws.services.kinesis.producer.KinesisProducerConfiguration producerConfig) Creates a KinesisProducer.
void	initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
void	invoke(OUT value, org.apache.flink.streaming.api.functions.sink.SinkFunction.Context context)
void	open(org.apache.flink.configuration.Configuration parameters)
void	setCustomPartitioner(KinesisPartitioner<OUT> partitioner)
void	setDefaultPartition(String defaultPartition) Set default partition id.
void	setDefaultStream(String defaultStream) Set a default stream name.
void	setFailOnError(boolean failOnError) If set to true, the producer will immediately fail with an exception on any error.
void	setQueueLimit(int queueLimit) The KinesisProducer holds an unbounded queue internally.
void	snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.functions.sink.SinkFunction

invoke

Field Detail

KINESIS_PRODUCER_METRIC_GROUP

public static final String KINESIS_PRODUCER_METRIC_GROUP

See Also:

Constant Field Values

METRIC_BACKPRESSURE_CYCLES

public static final String METRIC_BACKPRESSURE_CYCLES

See Also:

Constant Field Values

METRIC_OUTSTANDING_RECORDS_COUNT

public static final String METRIC_OUTSTANDING_RECORDS_COUNT

See Also:

Constant Field Values

Constructor Detail

FlinkKinesisProducer

public FlinkKinesisProducer(org.apache.flink.api.common.serialization.SerializationSchema<OUT> schema,
                            Properties configProps)

Create a new FlinkKinesisProducer. This is a constructor supporting Flink's SerializationSchema.

Parameters:

schema - Serialization schema for the data type

configProps - The properties used to configure KinesisProducer, including AWS credentials and AWS region

FlinkKinesisProducer

public FlinkKinesisProducer(KinesisSerializationSchema<OUT> schema,
                            Properties configProps)

Create a new FlinkKinesisProducer. This is a constructor supporting KinesisSerializationSchema.

Parameters:

schema - Kinesis serialization schema for the data type

configProps - The properties used to configure KinesisProducer, including AWS credentials and AWS region

Method Detail

setFailOnError

public void setFailOnError(boolean failOnError)

If set to true, the producer will immediately fail with an exception on any error. Otherwise, the errors are logged and the producer goes on.

Parameters:

failOnError - Error behavior flag

setQueueLimit

public void setQueueLimit(int queueLimit)

The KinesisProducer holds an unbounded queue internally. To avoid memory problems under high loads, a limit can be employed above which the internal queue will be flushed, thereby applying backpressure.

Parameters:

queueLimit - The maximum length of the internal queue before backpressuring

setDefaultStream

public void setDefaultStream(String defaultStream)

Set a default stream name.

Parameters:

defaultStream - Name of the default Kinesis stream

setDefaultPartition

public void setDefaultPartition(String defaultPartition)

Set default partition id.

Parameters:

defaultPartition - Name of the default partition

setCustomPartitioner

public void setCustomPartitioner(KinesisPartitioner<OUT> partitioner)

open

public void open(org.apache.flink.configuration.Configuration parameters)
          throws Exception

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

invoke

public void invoke(OUT value,
                   org.apache.flink.streaming.api.functions.sink.SinkFunction.Context context)
            throws Exception

Specified by:

invoke in interface org.apache.flink.streaming.api.functions.sink.SinkFunction<OUT>

Throws:

Exception

close

public void close()
           throws Exception

Specified by:

close in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

close in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

initializeState

public void initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
                     throws Exception

Specified by:

initializeState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Throws:

Exception

snapshotState

public void snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)
                   throws Exception

Specified by:

snapshotState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Throws:

Exception

getKinesisProducer

@VisibleForTesting
protected com.amazonaws.services.kinesis.producer.KinesisProducer getKinesisProducer(com.amazonaws.services.kinesis.producer.KinesisProducerConfiguration producerConfig)

Creates a KinesisProducer. Exposed so that tests can inject mock producers easily.