Package org.apache.flink.connector.kafka.source

Class KafkaSource<OUT>

java.lang.Object
org.apache.flink.connector.kafka.source.KafkaSource<OUT>
Type Parameters:
OUT - the output type of the source.
All Implemented Interfaces:
Serializable, org.apache.flink.api.connector.source.Source<OUT,KafkaPartitionSplit,KafkaSourceEnumState>, org.apache.flink.api.connector.source.SourceReaderFactory<OUT,KafkaPartitionSplit>, org.apache.flink.api.java.typeutils.ResultTypeQueryable<OUT>, org.apache.flink.streaming.api.lineage.LineageVertexProvider
@PublicEvolving public class KafkaSource<OUT> extends Object implements org.apache.flink.streaming.api.lineage.LineageVertexProvider, org.apache.flink.api.connector.source.Source<OUT,KafkaPartitionSplit,KafkaSourceEnumState>, org.apache.flink.api.java.typeutils.ResultTypeQueryable<OUT>
The Source implementation of Kafka. Please use a KafkaSourceBuilder to construct a KafkaSource. The following example shows how to create a KafkaSource emitting records of String type. 

 KafkaSource<String> source = KafkaSource
     .<String>builder()
     .setBootstrapServers(KafkaSourceTestEnv.brokerConnectionStrings)
     .setGroupId("MyGroup")
     .setTopics(Arrays.asList(TOPIC1, TOPIC2))
     .setDeserializer(new TestingKafkaRecordDeserializationSchema())
     .setStartingOffsets(OffsetsInitializer.earliest())
     .build();

KafkaSourceEnumerator only supports adding new splits and not removing splits in split discovery.

See KafkaSourceBuilder for more details on how to configure this source.

See Also:

  • Method Details

    • builder

      public static <OUT> KafkaSourceBuilder<OUT> builder()
      Get a kafkaSourceBuilder to build a KafkaSource.
      Returns:
      a Kafka source builder.

    • getBoundedness

      public org.apache.flink.api.connector.source.Boundedness getBoundedness()
      Specified by:
      getBoundedness in interface org.apache.flink.api.connector.source.Source<OUT,KafkaPartitionSplit,KafkaSourceEnumState>

    • createReader

      @Internal public org.apache.flink.api.connector.source.SourceReader<OUT,KafkaPartitionSplit> createReader(org.apache.flink.api.connector.source.SourceReaderContext readerContext) throws Exception
      Specified by:
      createReader in interface org.apache.flink.api.connector.source.SourceReaderFactory<OUT,KafkaPartitionSplit>
      Throws:
      Exception

    • createEnumerator

      @Internal public org.apache.flink.api.connector.source.SplitEnumerator<KafkaPartitionSplit,KafkaSourceEnumState> createEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<KafkaPartitionSplit> enumContext)
      Specified by:
      createEnumerator in interface org.apache.flink.api.connector.source.Source<OUT,KafkaPartitionSplit,KafkaSourceEnumState>

    • restoreEnumerator

      @Internal public org.apache.flink.api.connector.source.SplitEnumerator<KafkaPartitionSplit,KafkaSourceEnumState> restoreEnumerator(org.apache.flink.api.connector.source.SplitEnumeratorContext<KafkaPartitionSplit> enumContext, KafkaSourceEnumState checkpoint) throws IOException
      Specified by:
      restoreEnumerator in interface org.apache.flink.api.connector.source.Source<OUT,KafkaPartitionSplit,KafkaSourceEnumState>
      Throws:
      IOException

    • getSplitSerializer

      @Internal public org.apache.flink.core.io.SimpleVersionedSerializer<KafkaPartitionSplit> getSplitSerializer()
      Specified by:
      getSplitSerializer in interface org.apache.flink.api.connector.source.Source<OUT,KafkaPartitionSplit,KafkaSourceEnumState>

    • getEnumeratorCheckpointSerializer

      @Internal public org.apache.flink.core.io.SimpleVersionedSerializer<KafkaSourceEnumState> getEnumeratorCheckpointSerializer()
      Specified by:
      getEnumeratorCheckpointSerializer in interface org.apache.flink.api.connector.source.Source<OUT,KafkaPartitionSplit,KafkaSourceEnumState>

    • getProducedType

      public org.apache.flink.api.common.typeinfo.TypeInformation<OUT> getProducedType()
      Specified by:
      getProducedType in interface org.apache.flink.api.java.typeutils.ResultTypeQueryable<OUT>

    • getLineageVertex

      public org.apache.flink.streaming.api.lineage.SourceLineageVertex getLineageVertex()
      Specified by:
      getLineageVertex in interface org.apache.flink.streaming.api.lineage.LineageVertexProvider