Package org.apache.flink.connector.kafka.source.enumerator.subscriber

Interface KafkaSubscriber

All Superinterfaces:
Serializable
@PublicEvolving public interface KafkaSubscriber extends Serializable
Kafka consumer allows a few different ways to consume from the topics, including:
  1. Subscribe from a collection of topics.
  2. Subscribe to a topic pattern using Java Regex.
  3. Assign specific partitions.

The KafkaSubscriber provides a unified interface for the Kafka source to support all these three types of subscribing mode.

When implementing a subscriber, KafkaDatasetIdentifierProvider can be implemented to provide lineage metadata with source topics.

  • Method Details

    • getSubscribedTopicPartitions

      Set<org.apache.kafka.common.TopicPartition> getSubscribedTopicPartitions(org.apache.kafka.clients.admin.AdminClient adminClient)
      Get a set of subscribed TopicPartitions.
      Parameters:
      adminClient - The admin client used to retrieve subscribed topic partitions.
      Returns:
      A set of subscribed TopicPartitions

    • getTopicListSubscriber

      static KafkaSubscriber getTopicListSubscriber(List<String> topics)

    • getTopicPatternSubscriber

      static KafkaSubscriber getTopicPatternSubscriber(Pattern topicPattern)

    • getPartitionSetSubscriber

      static KafkaSubscriber getPartitionSetSubscriber(Set<org.apache.kafka.common.TopicPartition> partitions)