Package org.apache.flink.connector.kafka.sink

Class KafkaSink<IN>

java.lang.Object
org.apache.flink.connector.kafka.sink.KafkaSink<IN>
Type Parameters:
IN - type of the records written to Kafka
All Implemented Interfaces:
Serializable, org.apache.flink.api.connector.sink2.Sink<IN>, org.apache.flink.api.connector.sink2.StatefulSink<IN,org.apache.flink.connector.kafka.sink.KafkaWriterState>, org.apache.flink.api.connector.sink2.SupportsCommitter<org.apache.flink.connector.kafka.sink.KafkaCommittable>, org.apache.flink.api.connector.sink2.SupportsWriterState<IN,org.apache.flink.connector.kafka.sink.KafkaWriterState>, org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink<IN,org.apache.flink.connector.kafka.sink.KafkaCommittable>, TwoPhaseCommittingStatefulSink<IN,org.apache.flink.connector.kafka.sink.KafkaWriterState,org.apache.flink.connector.kafka.sink.KafkaCommittable>, org.apache.flink.streaming.api.lineage.LineageVertexProvider
@PublicEvolving public class KafkaSink<IN> extends Object implements org.apache.flink.streaming.api.lineage.LineageVertexProvider, TwoPhaseCommittingStatefulSink<IN,org.apache.flink.connector.kafka.sink.KafkaWriterState,org.apache.flink.connector.kafka.sink.KafkaCommittable>
Flink Sink to produce data into a Kafka topic. The sink supports all delivery guarantees described by DeliveryGuarantee.
  • DeliveryGuarantee.NONE does not provide any guarantees: messages may be lost in case of issues on the Kafka broker and messages may be duplicated in case of a Flink failure.
  • DeliveryGuarantee.AT_LEAST_ONCE the sink will wait for all outstanding records in the Kafka buffers to be acknowledged by the Kafka producer on a checkpoint. No messages will be lost in case of any issue with the Kafka brokers but messages may be duplicated when Flink restarts.
  • DeliveryGuarantee.EXACTLY_ONCE: In this mode the KafkaSink will write all messages in a Kafka transaction that will be committed to Kafka on a checkpoint. Thus, if the consumer reads only committed data (see Kafka consumer config isolation.level), no duplicates will be seen in case of a Flink restart. However, this delays record writing effectively until a checkpoint is written, so adjust the checkpoint duration accordingly. Please ensure that you use unique transactionalIdPrefixs across your applications running on the same Kafka cluster such that multiple running jobs do not interfere in their transactions! Additionally, it is highly recommended to tweak Kafka transaction timeout (link) >> maximum checkpoint duration + maximum restart duration or data loss may happen when Kafka expires an uncommitted transaction.
    • See Also:

    • Nested Class Summary

      Nested classes/interfaces inherited from interface org.apache.flink.api.connector.sink2.Sink

      org.apache.flink.api.connector.sink2.Sink.InitContext, org.apache.flink.api.connector.sink2.Sink.InitContextWrapper

      Nested classes/interfaces inherited from interface org.apache.flink.api.connector.sink2.StatefulSink

      org.apache.flink.api.connector.sink2.StatefulSink.StatefulSinkWriter<InputT,WriterStateT>, org.apache.flink.api.connector.sink2.StatefulSink.WithCompatibleState

      Nested classes/interfaces inherited from interface org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink

      org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink.PrecommittingSinkWriter<InputT,CommT>

      Nested classes/interfaces inherited from interface org.apache.flink.connector.kafka.sink.TwoPhaseCommittingStatefulSink

      TwoPhaseCommittingStatefulSink.PrecommittingStatefulSinkWriter<InputT,WriterStateT,CommT>

    • Method Summary

      Modifier and Type
      Method
      Description
      static <IN> KafkaSinkBuilder<IN>
      builder()
      Create a KafkaSinkBuilder to construct a new KafkaSink.
      org.apache.flink.api.connector.sink2.Committer<org.apache.flink.connector.kafka.sink.KafkaCommittable>
      createCommitter()
       
      org.apache.flink.connector.kafka.sink.KafkaWriter<IN>
      createWriter(org.apache.flink.api.connector.sink2.Sink.InitContext context)
       
      org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.flink.connector.kafka.sink.KafkaCommittable>
      getCommittableSerializer()
       
      protected Properties
      getKafkaProducerConfig()
       
      org.apache.flink.streaming.api.lineage.LineageVertex
      getLineageVertex()
       
      org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.flink.connector.kafka.sink.KafkaWriterState>
      getWriterStateSerializer()
       
      org.apache.flink.connector.kafka.sink.KafkaWriter<IN>
      restoreWriter(org.apache.flink.api.connector.sink2.Sink.InitContext context, Collection<org.apache.flink.connector.kafka.sink.KafkaWriterState> recoveredState)
       

      Methods inherited from class java.lang.Object

      clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

      Methods inherited from interface org.apache.flink.api.connector.sink2.Sink

      createWriter

      Methods inherited from interface org.apache.flink.api.connector.sink2.StatefulSink

      restoreWriter

      Methods inherited from interface org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink

      createCommitter

    • Method Details

      • builder

        public static <IN> KafkaSinkBuilder<IN> builder()
        Create a KafkaSinkBuilder to construct a new KafkaSink.
        Type Parameters:
        IN - type of incoming records
        Returns:
        KafkaSinkBuilder

      • createCommitter

        @Internal public org.apache.flink.api.connector.sink2.Committer<org.apache.flink.connector.kafka.sink.KafkaCommittable> createCommitter() throws IOException
        Specified by:
        createCommitter in interface org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink<IN,org.apache.flink.connector.kafka.sink.KafkaCommittable>
        Throws:
        IOException

      • getCommittableSerializer

        @Internal public org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.flink.connector.kafka.sink.KafkaCommittable> getCommittableSerializer()
        Specified by:
        getCommittableSerializer in interface org.apache.flink.api.connector.sink2.SupportsCommitter<IN>

      • createWriter

        @Internal public org.apache.flink.connector.kafka.sink.KafkaWriter<IN> createWriter(org.apache.flink.api.connector.sink2.Sink.InitContext context) throws IOException
        Specified by:
        createWriter in interface org.apache.flink.api.connector.sink2.Sink<IN>
        Specified by:
        createWriter in interface TwoPhaseCommittingStatefulSink<IN,org.apache.flink.connector.kafka.sink.KafkaWriterState,org.apache.flink.connector.kafka.sink.KafkaCommittable>
        Throws:
        IOException

      • restoreWriter

        @Internal public org.apache.flink.connector.kafka.sink.KafkaWriter<IN> restoreWriter(org.apache.flink.api.connector.sink2.Sink.InitContext context, Collection<org.apache.flink.connector.kafka.sink.KafkaWriterState> recoveredState) throws IOException
        Specified by:
        restoreWriter in interface org.apache.flink.api.connector.sink2.StatefulSink<IN,org.apache.flink.connector.kafka.sink.KafkaWriterState>
        Specified by:
        restoreWriter in interface TwoPhaseCommittingStatefulSink<IN,org.apache.flink.connector.kafka.sink.KafkaWriterState,org.apache.flink.connector.kafka.sink.KafkaCommittable>
        Throws:
        IOException

      • getWriterStateSerializer

        @Internal public org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.flink.connector.kafka.sink.KafkaWriterState> getWriterStateSerializer()
        Specified by:
        getWriterStateSerializer in interface org.apache.flink.api.connector.sink2.SupportsWriterState<IN,org.apache.flink.connector.kafka.sink.KafkaWriterState>

      • getKafkaProducerConfig

        @VisibleForTesting protected Properties getKafkaProducerConfig()

      • getLineageVertex

        public org.apache.flink.streaming.api.lineage.LineageVertex getLineageVertex()
        Specified by:
        getLineageVertex in interface org.apache.flink.streaming.api.lineage.LineageVertexProvider