Package org.apache.flink.streaming.connectors.kafka.internals

Class KafkaFetcher<T>

java.lang.Object

org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher<T,org.apache.kafka.common.TopicPartition>

org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher<T>

Type Parameters:

T - The type of elements produced by the fetcher.

Direct Known Subclasses:

KafkaShuffleFetcher

@Internal
public class KafkaFetcher<T>
extends AbstractFetcher<T,org.apache.kafka.common.TopicPartition>

A fetcher that fetches data from Kafka brokers via the Kafka consumer API.

Field Summary

Fields inherited from class org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher

checkpointLock, sourceContext, unassignedPartitionsQueue, watermarkOutput

Constructor Summary

Constructors

Constructor

Description

KafkaFetcher(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext, Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets, org.apache.flink.util.SerializedValue<org.apache.flink.api.common.eventtime.WatermarkStrategy<T>> watermarkStrategy, org.apache.flink.streaming.runtime.tasks.ProcessingTimeService processingTimeProvider, long autoWatermarkInterval, ClassLoader userCodeClassLoader, String taskNameWithSubtasks, KafkaDeserializationSchema<T> deserializer, Properties kafkaProperties, long pollTimeout, org.apache.flink.metrics.MetricGroup subtaskMetricGroup, org.apache.flink.metrics.MetricGroup consumerMetricGroup, boolean useMetrics)

Method Summary

Modifier and Type	Method	Description
void	cancel()
org.apache.kafka.common.TopicPartition	createKafkaPartitionHandle(KafkaTopicPartition partition)	Creates the Kafka version specific representation of the given topic partition.
protected void	doCommitInternalOffsetsToKafka(Map<KafkaTopicPartition,Long> offsets, KafkaCommitCallback commitCallback)
protected String	getFetcherName()	Gets the name of this fetcher, for thread naming and logging purposes.
protected void	partitionConsumerRecordsHandler(List<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>> partitionRecords, KafkaTopicPartitionState<T,org.apache.kafka.common.TopicPartition> partition)
void	runFetchLoop()

Methods inherited from class org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher

addDiscoveredPartitions, commitInternalOffsetsToKafka, emitRecordsWithTimestamps, snapshotCurrentState, subscribedPartitionStates

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

KafkaFetcher

public KafkaFetcher(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext,
                    Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets,
                    org.apache.flink.util.SerializedValue<org.apache.flink.api.common.eventtime.WatermarkStrategy<T>> watermarkStrategy,
                    org.apache.flink.streaming.runtime.tasks.ProcessingTimeService processingTimeProvider,
                    long autoWatermarkInterval,
                    ClassLoader userCodeClassLoader,
                    String taskNameWithSubtasks,
                    KafkaDeserializationSchema<T> deserializer,
                    Properties kafkaProperties,
                    long pollTimeout,
                    org.apache.flink.metrics.MetricGroup subtaskMetricGroup,
                    org.apache.flink.metrics.MetricGroup consumerMetricGroup,
                    boolean useMetrics)
             throws Exception

Throws:

Exception

Method Detail

runFetchLoop

public void runFetchLoop()
                  throws Exception

Specified by:

runFetchLoop in class AbstractFetcher<T,org.apache.kafka.common.TopicPartition>

Throws:

Exception

cancel

public void cancel()

Specified by:

cancel in class AbstractFetcher<T,org.apache.kafka.common.TopicPartition>

getFetcherName

protected String getFetcherName()

Gets the name of this fetcher, for thread naming and logging purposes.

partitionConsumerRecordsHandler

protected void partitionConsumerRecordsHandler(List<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>> partitionRecords,
                                               KafkaTopicPartitionState<T,org.apache.kafka.common.TopicPartition> partition)
                                        throws Exception

Throws:

Exception

createKafkaPartitionHandle

public org.apache.kafka.common.TopicPartition createKafkaPartitionHandle(KafkaTopicPartition partition)

Description copied from class: AbstractFetcher

Creates the Kafka version specific representation of the given topic partition.

Specified by:

createKafkaPartitionHandle in class AbstractFetcher<T,org.apache.kafka.common.TopicPartition>

Parameters:

partition - The Flink representation of the Kafka topic partition.

Returns:

The version-specific Kafka representation of the Kafka topic partition.

doCommitInternalOffsetsToKafka

protected void doCommitInternalOffsetsToKafka(Map<KafkaTopicPartition,Long> offsets,
                                              @Nonnull
                                              KafkaCommitCallback commitCallback)
                                       throws Exception

Specified by:

doCommitInternalOffsetsToKafka in class AbstractFetcher<T,org.apache.kafka.common.TopicPartition>

Throws:

Exception