Package org.apache.flink.streaming.connectors.kafka

Class FlinkKafkaConsumer<T>

java.lang.Object
org.apache.flink.api.common.functions.AbstractRichFunction
org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction<T>
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase<T>
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer<T>
All Implemented Interfaces:
Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.functions.source.ParallelSourceFunction<T>, org.apache.flink.streaming.api.functions.source.SourceFunction<T>
Direct Known Subclasses:
FlinkKafkaShuffleConsumer
@PublicEvolving @Deprecated public class FlinkKafkaConsumer<T> extends FlinkKafkaConsumerBase<T>
Deprecated.
The Flink Kafka Consumer is a streaming data source that pulls a parallel data stream from Apache Kafka. The consumer can run in multiple parallel instances, each of which will pull data from one or more Kafka partitions.

The Flink Kafka Consumer participates in checkpointing and guarantees that no data is lost during a failure, and that the computation processes elements "exactly once". (Note: These guarantees naturally assume that Kafka itself does not lose any data.)

Please note that Flink snapshots the offsets internally as part of its distributed checkpoints. The offsets committed to Kafka are only to bring the outside view of progress in sync with Flink's view of the progress. That way, monitoring and other jobs can get a view of how far the Flink Kafka consumer has consumed a topic.

Please refer to Kafka's documentation for the available configuration properties: http://kafka.apache.org/documentation.html#newconsumerconfigs

See Also:

  • Field Details

    • KEY_POLL_TIMEOUT

      public static final String KEY_POLL_TIMEOUT
      Deprecated.
      Configuration key to change the polling timeout. *
      See Also:

    • DEFAULT_POLL_TIMEOUT

      public static final long DEFAULT_POLL_TIMEOUT
      Deprecated.
      From Kafka's Javadoc: The time, in milliseconds, spent waiting in poll if data is not available. If 0, returns immediately with any records that are available now.
      See Also:

    • properties

      protected final Properties properties
      Deprecated.
      User-supplied properties for Kafka. *

    • pollTimeout

      protected final long pollTimeout
      Deprecated.
      From Kafka's Javadoc: The time, in milliseconds, spent waiting in poll if data is not available. If 0, returns immediately with any records that are available now

  • Constructor Details

    • FlinkKafkaConsumer

      public FlinkKafkaConsumer(String topic, org.apache.flink.api.common.serialization.DeserializationSchema<T> valueDeserializer, Properties props)
      Deprecated.
      Creates a new Kafka streaming source consumer.
      Parameters:
      topic - The name of the topic that should be consumed.
      valueDeserializer - The de-/serializer used to convert between Kafka's byte messages and Flink's objects.
      props -

    • FlinkKafkaConsumer

      public FlinkKafkaConsumer(String topic, KafkaDeserializationSchema<T> deserializer, Properties props)
      Deprecated.
      Creates a new Kafka streaming source consumer.

      This constructor allows passing a for reading key/value pairs, offsets, and topic names from Kafka.

      Parameters:
      topic - The name of the topic that should be consumed.
      deserializer - The keyed de-/serializer used to convert between Kafka's byte messages and Flink's objects.
      props -

    • FlinkKafkaConsumer

      public FlinkKafkaConsumer(List<String> topics, org.apache.flink.api.common.serialization.DeserializationSchema<T> deserializer, Properties props)
      Deprecated.
      Creates a new Kafka streaming source consumer.

      This constructor allows passing multiple topics to the consumer.

      Parameters:
      topics - The Kafka topics to read from.
      deserializer - The de-/serializer used to convert between Kafka's byte messages and Flink's objects.
      props -

    • FlinkKafkaConsumer

      public FlinkKafkaConsumer(List<String> topics, KafkaDeserializationSchema<T> deserializer, Properties props)
      Deprecated.
      Creates a new Kafka streaming source consumer.

      This constructor allows passing multiple topics and a key/value deserialization schema.

      Parameters:
      topics - The Kafka topics to read from.
      deserializer - The keyed de-/serializer used to convert between Kafka's byte messages and Flink's objects.
      props -

    • FlinkKafkaConsumer

      public FlinkKafkaConsumer(Pattern subscriptionPattern, org.apache.flink.api.common.serialization.DeserializationSchema<T> valueDeserializer, Properties props)
      Deprecated.
      Creates a new Kafka streaming source consumer. Use this constructor to subscribe to multiple topics based on a regular expression pattern.

      If partition discovery is enabled (by setting a non-negative value for FlinkKafkaConsumerBase.KEY_PARTITION_DISCOVERY_INTERVAL_MILLIS in the properties), topics with names matching the pattern will also be subscribed to as they are created on the fly.

      Parameters:
      subscriptionPattern - The regular expression for a pattern of topic names to subscribe to.
      valueDeserializer - The de-/serializer used to convert between Kafka's byte messages and Flink's objects.
      props -

    • FlinkKafkaConsumer

      public FlinkKafkaConsumer(Pattern subscriptionPattern, KafkaDeserializationSchema<T> deserializer, Properties props)
      Deprecated.
      Creates a new Kafka streaming source consumer. Use this constructor to subscribe to multiple topics based on a regular expression pattern.

      If partition discovery is enabled (by setting a non-negative value for FlinkKafkaConsumerBase.KEY_PARTITION_DISCOVERY_INTERVAL_MILLIS in the properties), topics with names matching the pattern will also be subscribed to as they are created on the fly.

      This constructor allows passing a for reading key/value pairs, offsets, and topic names from Kafka.

      Parameters:
      subscriptionPattern - The regular expression for a pattern of topic names to subscribe to.
      deserializer - The keyed de-/serializer used to convert between Kafka's byte messages and Flink's objects.
      props -

  • Method Details

    • createFetcher

      protected AbstractFetcher<T,?> createFetcher(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext, Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets, org.apache.flink.util.SerializedValue<org.apache.flink.api.common.eventtime.WatermarkStrategy<T>> watermarkStrategy, org.apache.flink.streaming.api.operators.StreamingRuntimeContext runtimeContext, OffsetCommitMode offsetCommitMode, org.apache.flink.metrics.MetricGroup consumerMetricGroup, boolean useMetrics) throws Exception
      Deprecated.
      Description copied from class: FlinkKafkaConsumerBase
      Creates the fetcher that connect to the Kafka brokers, pulls data, deserialized the data, and emits it into the data streams.
      Specified by:
      createFetcher in class FlinkKafkaConsumerBase<T>
      Parameters:
      sourceContext - The source context to emit data to.
      assignedPartitionsWithInitialOffsets - The set of partitions that this subtask should handle, with their start offsets.
      watermarkStrategy - Optional, a serialized WatermarkStrategy.
      runtimeContext - The task's runtime context.
      Returns:
      The instantiated fetcher
      Throws:
      Exception - The method should forward exceptions

    • createPartitionDiscoverer

      protected AbstractPartitionDiscoverer createPartitionDiscoverer(KafkaTopicsDescriptor topicsDescriptor, int indexOfThisSubtask, int numParallelSubtasks)
      Deprecated.
      Description copied from class: FlinkKafkaConsumerBase
      Creates the partition discoverer that is used to find new partitions for this subtask.
      Specified by:
      createPartitionDiscoverer in class FlinkKafkaConsumerBase<T>
      Parameters:
      topicsDescriptor - Descriptor that describes whether we are discovering partitions for fixed topics or a topic pattern.
      indexOfThisSubtask - The index of this consumer subtask.
      numParallelSubtasks - The total number of parallel consumer subtasks.
      Returns:
      The instantiated partition discoverer

    • fetchOffsetsWithTimestamp

      protected Map<KafkaTopicPartition,Long> fetchOffsetsWithTimestamp(Collection<KafkaTopicPartition> partitions, long timestamp)
      Deprecated.
      Specified by:
      fetchOffsetsWithTimestamp in class FlinkKafkaConsumerBase<T>

    • getIsAutoCommitEnabled

      protected boolean getIsAutoCommitEnabled()
      Deprecated.
      Specified by:
      getIsAutoCommitEnabled in class FlinkKafkaConsumerBase<T>