org.apache.flink.streaming.connectors.kafka

Class FlinkKafkaProducer<IN>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.sink.RichSinkFunction<IN>

org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer<IN>

Type Parameters:

IN - Type of the messages to write into Kafka.

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.streaming.api.functions.sink.SinkFunction<IN>

Direct Known Subclasses:

KafkaSink

public class FlinkKafkaProducer<IN>
extends org.apache.flink.streaming.api.functions.sink.RichSinkFunction<IN>

Flink Sink to produce data into a Kafka topic. Please note that this producer does not have any reliability guarantees.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
FlinkKafkaProducer(String topicId, org.apache.flink.streaming.util.serialization.SerializationSchema<IN,byte[]> serializationSchema, Properties producerConfig) Creates a FlinkKafkaProducer for a given topic.
FlinkKafkaProducer(String topicId, org.apache.flink.streaming.util.serialization.SerializationSchema<IN,byte[]> serializationSchema, Properties producerConfig, KafkaPartitioner customPartitioner) The main constructor for creating a FlinkKafkaProducer.
FlinkKafkaProducer(String brokerList, String topicId, org.apache.flink.streaming.util.serialization.SerializationSchema<IN,byte[]> serializationSchema) Creates a FlinkKafkaProducer for a given topic.

Method Summary

Modifier and Type	Method and Description
void	close()
static Properties	getPropertiesFromBrokerList(String brokerList)
void	invoke(IN next) Called when new data arrives to the sink, and forwards it to Kafka.
void	open(org.apache.flink.configuration.Configuration configuration) Initializes the connection to Kafka.
void	setLogFailuresOnly(boolean logFailuresOnly) Defines whether the producer should fail on errors, or only log them.

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FlinkKafkaProducer

public FlinkKafkaProducer(String brokerList,
                          String topicId,
                          org.apache.flink.streaming.util.serialization.SerializationSchema<IN,byte[]> serializationSchema)

Creates a FlinkKafkaProducer for a given topic. The sink produces its input to the topic.

Parameters:

brokerList - Comma separated addresses of the brokers

topicId - ID of the Kafka topic.

serializationSchema - User defined serialization schema.

FlinkKafkaProducer

public FlinkKafkaProducer(String topicId,
                          org.apache.flink.streaming.util.serialization.SerializationSchema<IN,byte[]> serializationSchema,
                          Properties producerConfig)

Creates a FlinkKafkaProducer for a given topic. The sink produces its input to the topic.

Parameters:

topicId - ID of the Kafka topic.

serializationSchema - User defined serialization schema.

producerConfig - Properties with the producer configuration.

FlinkKafkaProducer

public FlinkKafkaProducer(String topicId,
                          org.apache.flink.streaming.util.serialization.SerializationSchema<IN,byte[]> serializationSchema,
                          Properties producerConfig,
                          KafkaPartitioner customPartitioner)

The main constructor for creating a FlinkKafkaProducer.

Parameters:

topicId - The topic to write data to

serializationSchema - A serializable serialization schema for turning user objects into a kafka-consumable byte[]

producerConfig - Configuration properties for the KafkaProducer. 'bootstrap.servers.' is the only required argument.

customPartitioner - A serializable partitioner for assining messages to Kafka partitions.

Method Detail

setLogFailuresOnly

public void setLogFailuresOnly(boolean logFailuresOnly)

Defines whether the producer should fail on errors, or only log them. If this is set to true, then exceptions will be only logged, if set to false, exceptions will be eventually thrown and cause the streaming program to fail (and enter recovery).

Parameters:

logFailuresOnly - The flag to indicate logging-only on exceptions.

open

public void open(org.apache.flink.configuration.Configuration configuration)

Initializes the connection to Kafka.

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.api.common.functions.AbstractRichFunction

invoke

public void invoke(IN next)
            throws Exception

Called when new data arrives to the sink, and forwards it to Kafka.

Specified by:

invoke in interface org.apache.flink.streaming.api.functions.sink.SinkFunction<IN>

Specified by:

invoke in class org.apache.flink.streaming.api.functions.sink.RichSinkFunction<IN>

Parameters:

next - The incoming data

Throws:

Exception

close

public void close()
           throws Exception

Specified by:

close in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

close in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

getPropertiesFromBrokerList

public static Properties getPropertiesFromBrokerList(String brokerList)