Package org.apache.flink.streaming.connectors.kafka.internals

Class KafkaShuffleFetcher<T>

java.lang.Object

org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher<T,org.apache.kafka.common.TopicPartition>

org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher<T>

org.apache.flink.streaming.connectors.kafka.internals.KafkaShuffleFetcher<T>

@Internal
public class KafkaShuffleFetcher<T>
extends KafkaFetcher<T>

Fetch data from Kafka for Kafka Shuffle.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	KafkaShuffleFetcher.KafkaShuffleElement	An element in a KafkaShuffle.
static class	KafkaShuffleFetcher.KafkaShuffleElementDeserializer<T>	Deserializer for KafkaShuffleElement.
static class	KafkaShuffleFetcher.KafkaShuffleRecord<T>	One value with Type T in a KafkaShuffle.
static class	KafkaShuffleFetcher.KafkaShuffleWatermark	A watermark element in a KafkaShuffle.

Field Summary

Fields inherited from class org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher

checkpointLock, sourceContext, unassignedPartitionsQueue, watermarkOutput

Constructor Summary

Constructors

Constructor

Description

KafkaShuffleFetcher(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext, Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets, org.apache.flink.util.SerializedValue<org.apache.flink.api.common.eventtime.WatermarkStrategy<T>> watermarkStrategy, org.apache.flink.streaming.runtime.tasks.ProcessingTimeService processingTimeProvider, long autoWatermarkInterval, ClassLoader userCodeClassLoader, String taskNameWithSubtasks, KafkaDeserializationSchema<T> deserializer, Properties kafkaProperties, long pollTimeout, org.apache.flink.metrics.MetricGroup subtaskMetricGroup, org.apache.flink.metrics.MetricGroup consumerMetricGroup, boolean useMetrics, org.apache.flink.api.common.typeutils.TypeSerializer<T> typeSerializer, int producerParallelism)

Method Summary

Modifier and Type	Method	Description
protected String	getFetcherName()	Gets the name of this fetcher, for thread naming and logging purposes.
protected void	partitionConsumerRecordsHandler(List<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>> partitionRecords, KafkaTopicPartitionState<T,org.apache.kafka.common.TopicPartition> partition)

Methods inherited from class org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher

cancel, createKafkaPartitionHandle, doCommitInternalOffsetsToKafka, runFetchLoop

Methods inherited from class org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher

addDiscoveredPartitions, commitInternalOffsetsToKafka, emitRecordsWithTimestamps, snapshotCurrentState, subscribedPartitionStates

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

KafkaShuffleFetcher

public KafkaShuffleFetcher(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext,
                           Map<KafkaTopicPartition,Long> assignedPartitionsWithInitialOffsets,
                           org.apache.flink.util.SerializedValue<org.apache.flink.api.common.eventtime.WatermarkStrategy<T>> watermarkStrategy,
                           org.apache.flink.streaming.runtime.tasks.ProcessingTimeService processingTimeProvider,
                           long autoWatermarkInterval,
                           ClassLoader userCodeClassLoader,
                           String taskNameWithSubtasks,
                           KafkaDeserializationSchema<T> deserializer,
                           Properties kafkaProperties,
                           long pollTimeout,
                           org.apache.flink.metrics.MetricGroup subtaskMetricGroup,
                           org.apache.flink.metrics.MetricGroup consumerMetricGroup,
                           boolean useMetrics,
                           org.apache.flink.api.common.typeutils.TypeSerializer<T> typeSerializer,
                           int producerParallelism)
                    throws Exception

Throws:

Exception

Method Detail

getFetcherName

protected String getFetcherName()

Description copied from class: KafkaFetcher

Gets the name of this fetcher, for thread naming and logging purposes.

Overrides:

getFetcherName in class KafkaFetcher<T>

partitionConsumerRecordsHandler

protected void partitionConsumerRecordsHandler(List<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>> partitionRecords,
                                               KafkaTopicPartitionState<T,org.apache.kafka.common.TopicPartition> partition)
                                        throws Exception

Overrides:

partitionConsumerRecordsHandler in class KafkaFetcher<T>

Throws:

Exception