org.apache.flink.streaming.connectors.kafka.api

Class KafkaSource<OUT>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction<OUT>

org.apache.flink.streaming.api.functions.source.ConnectorSource<OUT>

org.apache.flink.streaming.connectors.kafka.api.KafkaSource<OUT>

Type Parameters:

OUT - Type of the messages on the topic.

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.api.java.typeutils.ResultTypeQueryable<OUT>, org.apache.flink.streaming.api.functions.source.ParallelSourceFunction<OUT>, org.apache.flink.streaming.api.functions.source.SourceFunction<OUT>

public class KafkaSource<OUT>
extends org.apache.flink.streaming.api.functions.source.ConnectorSource<OUT>

Source that listens to a Kafka topic using the high level Kafka API.

IMPORTANT: This source is not participating in the checkpointing procedure and hence gives no form of processing guarantees. Use the PersistentKafkaSource for a fault tolerant source that provides "exactly once" processing guarantees when used with checkpointing enabled.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T>

Field Summary

Fields inherited from class org.apache.flink.streaming.api.functions.source.ConnectorSource

schema

Constructor Summary

Constructors

Constructor and Description
KafkaSource(String zookeeperAddress, String topicId, org.apache.flink.streaming.util.serialization.DeserializationSchema<OUT> deserializationSchema) Creates a KafkaSource that consumes a topic.
KafkaSource(String zookeeperAddress, String topicId, org.apache.flink.streaming.util.serialization.DeserializationSchema<OUT> deserializationSchema, long zookeeperSyncTimeMillis) Creates a KafkaSource that consumes a topic.
KafkaSource(String zookeeperAddress, String topicId, String groupId, org.apache.flink.streaming.util.serialization.DeserializationSchema<OUT> deserializationSchema, long zookeeperSyncTimeMillis) Creates a KafkaSource that consumes a topic.
KafkaSource(String zookeeperAddress, String topicId, String groupId, org.apache.flink.streaming.util.serialization.DeserializationSchema<OUT> deserializationSchema, long zookeeperSyncTimeMillis, Properties customProperties) Creates a KafkaSource that consumes a topic.

Method Summary

Modifier and Type	Method and Description
void	cancel()
void	open(org.apache.flink.configuration.Configuration config)
void	run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<OUT> ctx)

Methods inherited from class org.apache.flink.streaming.api.functions.source.ConnectorSource

getProducedType

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

close, getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

KafkaSource

public KafkaSource(String zookeeperAddress,
                   String topicId,
                   String groupId,
                   org.apache.flink.streaming.util.serialization.DeserializationSchema<OUT> deserializationSchema,
                   long zookeeperSyncTimeMillis)

Creates a KafkaSource that consumes a topic.

Parameters:

zookeeperAddress - Address of the Zookeeper host (with port number).

topicId - ID of the Kafka topic.

groupId - ID of the consumer group.

deserializationSchema - User defined deserialization schema.

zookeeperSyncTimeMillis - Synchronization time with zookeeper.

KafkaSource

public KafkaSource(String zookeeperAddress,
                   String topicId,
                   String groupId,
                   org.apache.flink.streaming.util.serialization.DeserializationSchema<OUT> deserializationSchema,
                   long zookeeperSyncTimeMillis,
                   Properties customProperties)

Creates a KafkaSource that consumes a topic.

Parameters:

zookeeperAddress - Address of the Zookeeper host (with port number).

topicId - ID of the Kafka topic.

groupId - ID of the consumer group.

deserializationSchema - User defined deserialization schema.

zookeeperSyncTimeMillis - Synchronization time with zookeeper.

customProperties - Custom properties for Kafka

KafkaSource

public KafkaSource(String zookeeperAddress,
                   String topicId,
                   org.apache.flink.streaming.util.serialization.DeserializationSchema<OUT> deserializationSchema,
                   long zookeeperSyncTimeMillis)

Creates a KafkaSource that consumes a topic.

Parameters:

zookeeperAddress - Address of the Zookeeper host (with port number).

topicId - ID of the Kafka topic.

deserializationSchema - User defined deserialization schema.

zookeeperSyncTimeMillis - Synchronization time with zookeeper.

KafkaSource

public KafkaSource(String zookeeperAddress,
                   String topicId,
                   org.apache.flink.streaming.util.serialization.DeserializationSchema<OUT> deserializationSchema)

Creates a KafkaSource that consumes a topic.

Parameters:

zookeeperAddress - Address of the Zookeeper host (with port number).

topicId - ID of the Kafka topic.

deserializationSchema - User defined deserialization schema.

Method Detail

run

public void run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<OUT> ctx)
         throws Exception

Throws:

Exception

open

public void open(org.apache.flink.configuration.Configuration config)
          throws Exception

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

cancel

public void cancel()