org.apache.flink.streaming.connectors.kafka.api.persistent

Class PersistentKafkaSource<OUT>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction<OUT>

org.apache.flink.streaming.connectors.kafka.api.persistent.PersistentKafkaSource<OUT>

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.api.java.typeutils.ResultTypeQueryable<OUT>, org.apache.flink.streaming.api.checkpoint.CheckpointCommitter, org.apache.flink.streaming.api.checkpoint.Checkpointed<long[]>, org.apache.flink.streaming.api.checkpoint.CheckpointedAsynchronously<long[]>, org.apache.flink.streaming.api.functions.source.ParallelSourceFunction<OUT>, org.apache.flink.streaming.api.functions.source.SourceFunction<OUT>

public class PersistentKafkaSource<OUT>
extends org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction<OUT>
implements org.apache.flink.api.java.typeutils.ResultTypeQueryable<OUT>, org.apache.flink.streaming.api.checkpoint.CheckpointCommitter, org.apache.flink.streaming.api.checkpoint.CheckpointedAsynchronously<long[]>

Source for reading from Kafka using Flink Streaming Fault Tolerance. This source is updating the committed offset in Zookeeper based on the internal checkpointing of Flink. Note that the autocommit feature of Kafka needs to be disabled for using this source.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	PersistentKafkaSource.KafkaZKStringSerializer

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T>

Constructor Summary

Constructors

Constructor and Description
PersistentKafkaSource(String topicName, org.apache.flink.streaming.util.serialization.DeserializationSchema<OUT> deserializationSchema, kafka.consumer.ConsumerConfig consumerConfig) For the @param consumerConfig, specify at least the "groupid" and "zookeeper.connect" string.

Method Summary

Modifier and Type	Method and Description
void	cancel()
void	close()
void	commitCheckpoint(long checkpointId) Notification on completed checkpoints
static long	getOffset(org.I0Itec.zkclient.ZkClient zkClient, String groupId, String topic, int partition)
org.apache.flink.api.common.typeinfo.TypeInformation<OUT>	getProducedType()
void	open(org.apache.flink.configuration.Configuration parameters)
void	restoreState(long[] state)
void	run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<OUT> ctx)
protected void	setOffset(int partition, long offset)
static void	setOffset(org.I0Itec.zkclient.ZkClient zkClient, String groupId, String topic, int partition, long offset) This method's code is based on ZookeeperConsumerConnector.commitOffsetToZooKeeper()
long[]	snapshotState(long checkpointId, long checkpointTimestamp)

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PersistentKafkaSource

public PersistentKafkaSource(String topicName,
                             org.apache.flink.streaming.util.serialization.DeserializationSchema<OUT> deserializationSchema,
                             kafka.consumer.ConsumerConfig consumerConfig)

For the @param consumerConfig, specify at least the "groupid" and "zookeeper.connect" string. The config will be passed into the Kafka High Level Consumer. For a full list of possible values, check this out: https://kafka.apache.org/documentation.html#consumerconfigs

Method Detail

open

public void open(org.apache.flink.configuration.Configuration parameters)
          throws Exception

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

run

public void run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<OUT> ctx)
         throws Exception

Specified by:

run in interface org.apache.flink.streaming.api.functions.source.SourceFunction<OUT>

Throws:

Exception

cancel

public void cancel()

Specified by:

cancel in interface org.apache.flink.streaming.api.functions.source.SourceFunction<OUT>

close

public void close()

Specified by:

close in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

close in class org.apache.flink.api.common.functions.AbstractRichFunction

snapshotState

public long[] snapshotState(long checkpointId,
                            long checkpointTimestamp)
                     throws Exception

Specified by:

snapshotState in interface org.apache.flink.streaming.api.checkpoint.Checkpointed<long[]>

Throws:

Exception

restoreState

public void restoreState(long[] state)

Specified by:

restoreState in interface org.apache.flink.streaming.api.checkpoint.Checkpointed<long[]>

commitCheckpoint

public void commitCheckpoint(long checkpointId)

Notification on completed checkpoints

Specified by:

commitCheckpoint in interface org.apache.flink.streaming.api.checkpoint.CheckpointCommitter

Parameters:

checkpointId - The ID of the checkpoint that has been completed.

setOffset

protected void setOffset(int partition,
                         long offset)

setOffset

public static void setOffset(org.I0Itec.zkclient.ZkClient zkClient,
                             String groupId,
                             String topic,
                             int partition,
                             long offset)

This method's code is based on ZookeeperConsumerConnector.commitOffsetToZooKeeper()

getOffset

public static long getOffset(org.I0Itec.zkclient.ZkClient zkClient,
                             String groupId,
                             String topic,
                             int partition)

getProducedType

public org.apache.flink.api.common.typeinfo.TypeInformation<OUT> getProducedType()

Specified by:

getProducedType in interface org.apache.flink.api.java.typeutils.ResultTypeQueryable<OUT>