org.apache.flink.streaming.connectors.kafka.internals

Interface Fetcher

All Known Implementing Classes:

LegacyFetcher

public interface Fetcher

A fetcher pulls data from Kafka, from a fix set of partitions. The fetcher supports "seeking" inside the partitions, i.e., moving to a different offset.

Method Summary

Modifier and Type	Method and Description
void	close() Closes the fetcher.
<T> void	run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext, org.apache.flink.streaming.util.serialization.DeserializationSchema<T> valueDeserializer, long[] lastOffsets) Starts fetch data from Kafka and emitting it into the stream.
void	seek(org.apache.kafka.common.TopicPartition topicPartition, long offsetToRead) Set the next offset to read from for the given partition.
void	setPartitionsToRead(List<org.apache.kafka.common.TopicPartition> partitions) Set which partitions the fetcher should pull from.

Method Detail

setPartitionsToRead

void setPartitionsToRead(List<org.apache.kafka.common.TopicPartition> partitions)

Set which partitions the fetcher should pull from.

Parameters:

partitions - The list of partitions for a topic that the fetcher will pull from.

close

void close()
    throws IOException

Closes the fetcher. This will stop any operation in the run(SourceFunction.SourceContext, DeserializationSchema, long[]) method and eventually close underlying connections and release all resources.

Throws:

IOException

run

<T> void run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> sourceContext,
             org.apache.flink.streaming.util.serialization.DeserializationSchema<T> valueDeserializer,
             long[] lastOffsets)
      throws Exception

Starts fetch data from Kafka and emitting it into the stream.

To provide exactly once guarantees, the fetcher needs emit a record and update the update of the last consumed offset in one atomic operation:

 
 while (running) {
     T next = ...
     long offset = ...
     int partition = ...
     synchronized (sourceContext.getCheckpointLock()) {
         sourceContext.collect(next);
         lastOffsets[partition] = offset;
     }
 }
 

Type Parameters:

T - The type of elements produced by the fetcher and emitted to the source context.

Parameters:

sourceContext - The source context to emit elements to.

valueDeserializer - The deserializer to decode the raw values with.

lastOffsets - The array into which to store the offsets foe which elements are emitted.

Throws:

Exception

seek

void seek(org.apache.kafka.common.TopicPartition topicPartition,
          long offsetToRead)

Set the next offset to read from for the given partition. For example, if the partition i offset is set to n, the Fetcher's next result will be the message with offset=n.

Parameters:

topicPartition - The partition for which to seek the offset.

offsetToRead - To offset to seek to.