org.apache.flink.streaming.connectors.kafka

Class FlinkKafkaConsumer<T>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction<T>

org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer<T>

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>, Checkpointed<long[]>, CheckpointedAsynchronously<long[]>, CheckpointNotifier, ParallelSourceFunction<T>, SourceFunction<T>

Direct Known Subclasses:

FlinkKafkaConsumer081, FlinkKafkaConsumer082, PersistentKafkaSource

public class FlinkKafkaConsumer<T>
extends RichParallelSourceFunction<T>
implements CheckpointNotifier, CheckpointedAsynchronously<long[]>, org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>

The Flink Kafka Consumer is a streaming data source that pulls a parallel data stream from Apache Kafka. The consumer can run in multiple parallel instances, each of which will pull data from one or more Kafka partitions.

The Flink Kafka Consumer participates in checkpointing and guarantees that no data is lost during a failure, and that the computation processes elements "exactly once". (Note: These guarantees naturally assume that Kafka itself does not loose any data.)

To support a variety of Kafka brokers, protocol versions, and offset committing approaches, the Flink Kafka Consumer can be parametrized with a fetcher and an offset handler.

Fetcher

The fetcher is responsible to pull data from Kafka. Because Kafka has undergone a change in protocols and APIs, there are currently two fetchers available:

• FlinkKafkaConsumer.FetcherType.NEW_HIGH_LEVEL: A fetcher based on the new Kafka consumer API. This fetcher is generally more robust, but works only with later versions of Kafka (> 0.8.2).
• FlinkKafkaConsumer.FetcherType.LEGACY_LOW_LEVEL: A fetcher based on the old low-level consumer API. This fetcher is works also with older versions of Kafka (0.8.1). The fetcher interprets the old Kafka consumer properties, like:
  • socket.timeout.ms
  • socket.receive.buffer.bytes
  • fetch.message.max.bytes
  • auto.offset.reset with the values "latest", "earliest" (unlike 0.8.2 behavior)
  • fetch.wait.max.ms

Offset handler

Offsets whose records have been read and are checkpointed will be committed back to Kafka / ZooKeeper by the offset handler. In addition, the offset handler finds the point where the source initially starts reading from the stream, when the streaming job is started.

Currently, the source offers two different offset handlers exist:

• FlinkKafkaConsumer.OffsetStore.KAFKA: Use this offset handler when the Kafka brokers are managing the offsets, and hence offsets need to be committed the Kafka brokers, rather than to ZooKeeper. Note that this offset handler works only on new versions of Kafka (0.8.2.x +) and with the FlinkKafkaConsumer.FetcherType.NEW_HIGH_LEVEL fetcher.
• FlinkKafkaConsumer.OffsetStore.FLINK_ZOOKEEPER: Use this offset handler when the offsets are managed by ZooKeeper, as in older versions of Kafka (0.8.1.x)

Please note that Flink snapshots the offsets internally as part of its distributed checkpoints. The offsets committed to Kafka / ZooKeeper are only to bring the outside view of progress in sync with Flink's view of the progress. That way, monitoring and other jobs can get a view of how far the Flink Kafka consumer has consumed a topic.

NOTE: The implementation currently accesses partition metadata when the consumer is constructed. That means that the client that submits the program needs to be able to reach the Kafka brokers or ZooKeeper.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FlinkKafkaConsumer.FetcherType The fetcher type defines which code paths to use to pull data from teh Kafka broker.
static class	FlinkKafkaConsumer.OffsetStore The offset store defines how acknowledged offsets are committed back to Kafka.

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

SourceFunction.SourceContext<T>

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_GET_PARTITIONS_RETRIES Default number of retries for getting the partition info.
static String	GET_PARTITIONS_RETRIES_KEY Configuration key for the number of retries for getting the partition info
static int	MAX_NUM_PENDING_CHECKPOINTS The maximum number of pending non-committed checkpoints to track, to avoid memory leaks
static long	OFFSET_NOT_SET Magic number to define an unset offset.

Constructor Summary

Constructors

Constructor and Description
FlinkKafkaConsumer(String topic, DeserializationSchema<T> valueDeserializer, Properties props, FlinkKafkaConsumer.OffsetStore offsetStore, FlinkKafkaConsumer.FetcherType fetcherType) Creates a new Flink Kafka Consumer, using the given type of fetcher and offset handler.

Method Summary

Methods

Modifier and Type	Method and Description
protected static List<org.apache.kafka.common.TopicPartition>	assignPartitions(int[] partitions, String topicName, int numConsumers, int consumerIndex)
void	cancel()
void	close()
static List<org.apache.kafka.common.PartitionInfo>	getPartitionsForTopic(String topic, Properties properties) Send request to Kafka to get partitions for topic.
org.apache.flink.api.common.typeinfo.TypeInformation<T>	getProducedType()
void	notifyCheckpointComplete(long checkpointId)
void	open(org.apache.flink.configuration.Configuration parameters)
void	restoreState(long[] restoredOffsets)
void	run(SourceFunction.SourceContext<T> sourceContext)
long[]	snapshotState(long checkpointId, long checkpointTimestamp)
protected static void	validateZooKeeperConfig(Properties props) Validate the ZK configuration, checking for required parameters

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

OFFSET_NOT_SET

public static final long OFFSET_NOT_SET

Magic number to define an unset offset. Negative offsets are not used by Kafka (invalid), and we pick a number that is probably (hopefully) not used by Kafka as a magic number for anything else.

See Also:

Constant Field Values

MAX_NUM_PENDING_CHECKPOINTS

public static final int MAX_NUM_PENDING_CHECKPOINTS

The maximum number of pending non-committed checkpoints to track, to avoid memory leaks

See Also:

Constant Field Values

GET_PARTITIONS_RETRIES_KEY

public static final String GET_PARTITIONS_RETRIES_KEY

Configuration key for the number of retries for getting the partition info

See Also:

Constant Field Values

DEFAULT_GET_PARTITIONS_RETRIES

public static final int DEFAULT_GET_PARTITIONS_RETRIES

Default number of retries for getting the partition info. One retry means going through the full list of brokers

See Also:

Constant Field Values

Constructor Detail

FlinkKafkaConsumer

public FlinkKafkaConsumer(String topic,
                  DeserializationSchema<T> valueDeserializer,
                  Properties props,
                  FlinkKafkaConsumer.OffsetStore offsetStore,
                  FlinkKafkaConsumer.FetcherType fetcherType)

Creates a new Flink Kafka Consumer, using the given type of fetcher and offset handler.

To determine which kink of fetcher and offset handler to use, please refer to the docs at the beginnign of this class.

Parameters:

topic - The Kafka topic to read from.

valueDeserializer - The deserializer to turn raw byte messages into Java/Scala objects.

props - The properties that are used to configure both the fetcher and the offset handler.

offsetStore - The type of offset store to use (Kafka / ZooKeeper)

fetcherType - The type of fetcher to use (new high-level API, old low-level API).

Method Detail

open

public void open(org.apache.flink.configuration.Configuration parameters)
          throws Exception

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

run

public void run(SourceFunction.SourceContext<T> sourceContext)
         throws Exception

Specified by:

run in interface SourceFunction<T>

Throws:

Exception

cancel

public void cancel()

Specified by:

cancel in interface SourceFunction<T>

close

public void close()
           throws Exception

Specified by:

close in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

close in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

getProducedType

public org.apache.flink.api.common.typeinfo.TypeInformation<T> getProducedType()

Specified by:

getProducedType in interface org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>

snapshotState

public long[] snapshotState(long checkpointId,
                   long checkpointTimestamp)
                     throws Exception

Specified by:

snapshotState in interface Checkpointed<long[]>

Throws:

Exception

restoreState

public void restoreState(long[] restoredOffsets)

Specified by:

restoreState in interface Checkpointed<long[]>

notifyCheckpointComplete

public void notifyCheckpointComplete(long checkpointId)
                              throws Exception

Specified by:

notifyCheckpointComplete in interface CheckpointNotifier

Throws:

Exception

assignPartitions

protected static List<org.apache.kafka.common.TopicPartition> assignPartitions(int[] partitions,
                                                            String topicName,
                                                            int numConsumers,
                                                            int consumerIndex)

getPartitionsForTopic

public static List<org.apache.kafka.common.PartitionInfo> getPartitionsForTopic(String topic,
                                                                Properties properties)

Send request to Kafka to get partitions for topic.

Parameters:

topic - The name of the topic.

properties - The properties for the Kafka Consumer that is used to query the partitions for the topic.

validateZooKeeperConfig

protected static void validateZooKeeperConfig(Properties props)

Validate the ZK configuration, checking for required parameters

Parameters:

props - Properties to check