Package org.apache.flink.connector.kafka.source.reader.fetcher

Class KafkaSourceFetcherManager

java.lang.Object
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,KafkaPartitionSplit>
org.apache.flink.connector.base.source.reader.fetcher.SingleThreadFetcherManager<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,KafkaPartitionSplit>
org.apache.flink.connector.kafka.source.reader.fetcher.KafkaSourceFetcherManager
@Internal public class KafkaSourceFetcherManager extends org.apache.flink.connector.base.source.reader.fetcher.SingleThreadFetcherManager<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,KafkaPartitionSplit>
The SplitFetcherManager for Kafka source. This class is needed to help commit the offsets to Kafka using the KafkaConsumer inside the KafkaPartitionSplitReader.

  • Field Summary

    Fields inherited from class org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager

    fetchers

  • Constructor Summary

    Constructors
    Constructor
    Description
    KafkaSourceFetcherManager(org.apache.flink.connector.base.source.reader.synchronization.FutureCompletingBlockingQueue<org.apache.flink.connector.base.source.reader.RecordsWithSplitIds<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>>> elementsQueue, java.util.function.Supplier<org.apache.flink.connector.base.source.reader.splitreader.SplitReader<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,KafkaPartitionSplit>> splitReaderSupplier, java.util.function.Consumer<Collection<String>> splitFinishedHook)
    Creates a new SplitFetcherManager with a single I/O threads.

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    commitOffsets(Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> offsetsToCommit, org.apache.kafka.clients.consumer.OffsetCommitCallback callback)
     

    Methods inherited from class org.apache.flink.connector.base.source.reader.fetcher.SingleThreadFetcherManager

    addSplits, getRunningFetcher, removeSplits

    Methods inherited from class org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager

    checkErrors, close, createSplitFetcher, getNumAliveFetchers, getQueue, maybeShutdownFinishedFetchers, pauseOrResumeSplits, startFetcher

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • KafkaSourceFetcherManager

      public KafkaSourceFetcherManager(org.apache.flink.connector.base.source.reader.synchronization.FutureCompletingBlockingQueue<org.apache.flink.connector.base.source.reader.RecordsWithSplitIds<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>>> elementsQueue, java.util.function.Supplier<org.apache.flink.connector.base.source.reader.splitreader.SplitReader<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,KafkaPartitionSplit>> splitReaderSupplier, java.util.function.Consumer<Collection<String>> splitFinishedHook)
      Creates a new SplitFetcherManager with a single I/O threads.
      Parameters:
      elementsQueue - The queue that is used to hand over data from the I/O thread (the fetchers) to the reader (which emits the records and book-keeps the state. This must be the same queue instance that is also passed to the SourceReaderBase.
      splitReaderSupplier - The factory for the split reader that connects to the source system.
      splitFinishedHook - Hook for handling finished splits in split fetchers.

  • Method Details

    • commitOffsets

      public void commitOffsets(Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> offsetsToCommit, org.apache.kafka.clients.consumer.OffsetCommitCallback callback)