Package org.apache.flink.connector.kafka.sink

Class KafkaSink<IN>

  • Type Parameters:
    IN - type of the records written to Kafka
    All Implemented Interfaces:
    Serializable, org.apache.flink.api.connector.sink2.Sink<IN>, org.apache.flink.api.connector.sink2.StatefulSink<IN,​org.apache.flink.connector.kafka.sink.KafkaWriterState>, org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink<IN,​org.apache.flink.connector.kafka.sink.KafkaCommittable>
    @PublicEvolving
public class KafkaSink<IN>
extends Object
implements org.apache.flink.api.connector.sink2.StatefulSink<IN,​org.apache.flink.connector.kafka.sink.KafkaWriterState>, org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink<IN,​org.apache.flink.connector.kafka.sink.KafkaCommittable>
    Flink Sink to produce data into a Kafka topic. The sink supports all delivery guarantees described by DeliveryGuarantee.
  • DeliveryGuarantee.NONE does not provide any guarantees: messages may be lost in case of issues on the Kafka broker and messages may be duplicated in case of a Flink failure.
  • DeliveryGuarantee.AT_LEAST_ONCE the sink will wait for all outstanding records in the Kafka buffers to be acknowledged by the Kafka producer on a checkpoint. No messages will be lost in case of any issue with the Kafka brokers but messages may be duplicated when Flink restarts.
  • DeliveryGuarantee.EXACTLY_ONCE: In this mode the KafkaSink will write all messages in a Kafka transaction that will be committed to Kafka on a checkpoint. Thus, if the consumer reads only committed data (see Kafka consumer config isolation.level), no duplicates will be seen in case of a Flink restart. However, this delays record writing effectively until a checkpoint is written, so adjust the checkpoint duration accordingly. Please ensure that you use unique transactionalIdPrefixs across your applications running on the same Kafka cluster such that multiple running jobs do not interfere in their transactions! Additionally, it is highly recommended to tweak Kafka transaction timeout (link) >> maximum checkpoint duration + maximum restart duration or data loss may happen when Kafka expires an uncommitted transaction.
See Also:
on how to construct a KafkaSink, Serialized Form