Package org.apache.flink.connector.kafka.source.reader.fetcher

Class KafkaSourceFetcherManager

  • @Internal
public class KafkaSourceFetcherManager
extends org.apache.flink.connector.base.source.reader.fetcher.SingleThreadFetcherManager<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],​byte[]>,​KafkaPartitionSplit>
    The SplitFetcherManager for Kafka source. This class is needed to help commit the offsets to Kafka using the KafkaConsumer inside the KafkaPartitionSplitReader.

    • Field Summary

      • Fields inherited from class org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager

        fetchers

    • Constructor Summary

      Constructors 
      Constructor Description
      KafkaSourceFetcherManager​(org.apache.flink.connector.base.source.reader.synchronization.FutureCompletingBlockingQueue<org.apache.flink.connector.base.source.reader.RecordsWithSplitIds<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],​byte[]>>> elementsQueue, java.util.function.Supplier<org.apache.flink.connector.base.source.reader.splitreader.SplitReader<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],​byte[]>,​KafkaPartitionSplit>> splitReaderSupplier, java.util.function.Consumer<Collection<String>> splitFinishedHook)
      Creates a new SplitFetcherManager with a single I/O threads.

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void commitOffsets​(Map<org.apache.kafka.common.TopicPartition,​org.apache.kafka.clients.consumer.OffsetAndMetadata> offsetsToCommit, org.apache.kafka.clients.consumer.OffsetCommitCallback callback)  

      • Methods inherited from class org.apache.flink.connector.base.source.reader.fetcher.SingleThreadFetcherManager

        addSplits, getRunningFetcher

      • Methods inherited from class org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager

        checkErrors, close, createSplitFetcher, getNumAliveFetchers, maybeShutdownFinishedFetchers, pauseOrResumeSplits, startFetcher

    • Constructor Detail

      • KafkaSourceFetcherManager

        public KafkaSourceFetcherManager​(org.apache.flink.connector.base.source.reader.synchronization.FutureCompletingBlockingQueue<org.apache.flink.connector.base.source.reader.RecordsWithSplitIds<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],​byte[]>>> elementsQueue,
                                 java.util.function.Supplier<org.apache.flink.connector.base.source.reader.splitreader.SplitReader<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],​byte[]>,​KafkaPartitionSplit>> splitReaderSupplier,
                                 java.util.function.Consumer<Collection<String>> splitFinishedHook)
        Creates a new SplitFetcherManager with a single I/O threads.
        Parameters:
        elementsQueue - The queue that is used to hand over data from the I/O thread (the fetchers) to the reader (which emits the records and book-keeps the state. This must be the same queue instance that is also passed to the SourceReaderBase.
        splitReaderSupplier - The factory for the split reader that connects to the source system.
        splitFinishedHook - Hook for handling finished splits in split fetchers.

    • Method Detail

      • commitOffsets

        public void commitOffsets​(Map<org.apache.kafka.common.TopicPartition,​org.apache.kafka.clients.consumer.OffsetAndMetadata> offsetsToCommit,
                          org.apache.kafka.clients.consumer.OffsetCommitCallback callback)