Package org.apache.flink.connector.kafka.source.enumerator

Class KafkaSourceEnumerator

java.lang.Object

org.apache.flink.connector.kafka.source.enumerator.KafkaSourceEnumerator

All Implemented Interfaces:

AutoCloseable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.api.connector.source.SplitEnumerator<KafkaPartitionSplit,KafkaSourceEnumState>

@Internal
public class KafkaSourceEnumerator
extends Object
implements org.apache.flink.api.connector.source.SplitEnumerator<KafkaPartitionSplit,KafkaSourceEnumState>

The enumerator class for Kafka source.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	KafkaSourceEnumerator.PartitionOffsetsRetrieverImpl	The implementation for offsets retriever with a consumer and an admin client.

Constructor Summary

Constructors

Constructor	Description
KafkaSourceEnumerator(KafkaSubscriber subscriber, OffsetsInitializer startingOffsetInitializer, OffsetsInitializer stoppingOffsetInitializer, Properties properties, org.apache.flink.api.connector.source.SplitEnumeratorContext<KafkaPartitionSplit> context, org.apache.flink.api.connector.source.Boundedness boundedness)
KafkaSourceEnumerator(KafkaSubscriber subscriber, OffsetsInitializer startingOffsetInitializer, OffsetsInitializer stoppingOffsetInitializer, Properties properties, org.apache.flink.api.connector.source.SplitEnumeratorContext<KafkaPartitionSplit> context, org.apache.flink.api.connector.source.Boundedness boundedness, KafkaSourceEnumState kafkaSourceEnumState)

Method Summary

Modifier and Type	Method	Description
void	addReader(int subtaskId)
void	addSplitsBack(List<KafkaPartitionSplit> splits, int subtaskId)
void	close()
void	handleSplitRequest(int subtaskId, String requesterHostname)
KafkaSourceEnumState	snapshotState(long checkpointId)
void	start()	Start the enumerator.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted

Methods inherited from interface org.apache.flink.api.connector.source.SplitEnumerator

handleSourceEvent, notifyCheckpointComplete

Constructor Detail

KafkaSourceEnumerator

public KafkaSourceEnumerator(KafkaSubscriber subscriber,
                             OffsetsInitializer startingOffsetInitializer,
                             OffsetsInitializer stoppingOffsetInitializer,
                             Properties properties,
                             org.apache.flink.api.connector.source.SplitEnumeratorContext<KafkaPartitionSplit> context,
                             org.apache.flink.api.connector.source.Boundedness boundedness)

KafkaSourceEnumerator

public KafkaSourceEnumerator(KafkaSubscriber subscriber,
                             OffsetsInitializer startingOffsetInitializer,
                             OffsetsInitializer stoppingOffsetInitializer,
                             Properties properties,
                             org.apache.flink.api.connector.source.SplitEnumeratorContext<KafkaPartitionSplit> context,
                             org.apache.flink.api.connector.source.Boundedness boundedness,
                             KafkaSourceEnumState kafkaSourceEnumState)

Method Detail

start

public void start()

Start the enumerator.

Depending on partitionDiscoveryIntervalMs, the enumerator will trigger a one-time partition discovery, or schedule a callable for discover partitions periodically.

The invoking chain of partition discovery would be:

1. getSubscribedTopicPartitions() in worker thread
2. checkPartitionChanges(java.util.Set<org.apache.kafka.common.TopicPartition>, java.lang.Throwable) in coordinator thread
3. initializePartitionSplits(org.apache.flink.connector.kafka.source.enumerator.KafkaSourceEnumerator.PartitionChange) in worker thread
4. handlePartitionSplitChanges(org.apache.flink.connector.kafka.source.enumerator.KafkaSourceEnumerator.PartitionSplitChange, java.lang.Throwable) in coordinator thread

Specified by:

start in interface org.apache.flink.api.connector.source.SplitEnumerator<KafkaPartitionSplit,KafkaSourceEnumState>

handleSplitRequest

public void handleSplitRequest(int subtaskId,
                               @Nullable
                               String requesterHostname)

Specified by:

handleSplitRequest in interface org.apache.flink.api.connector.source.SplitEnumerator<KafkaPartitionSplit,KafkaSourceEnumState>

addSplitsBack

public void addSplitsBack(List<KafkaPartitionSplit> splits,
                          int subtaskId)

Specified by:

addSplitsBack in interface org.apache.flink.api.connector.source.SplitEnumerator<KafkaPartitionSplit,KafkaSourceEnumState>

addReader

public void addReader(int subtaskId)

Specified by:

addReader in interface org.apache.flink.api.connector.source.SplitEnumerator<KafkaPartitionSplit,KafkaSourceEnumState>

snapshotState

public KafkaSourceEnumState snapshotState(long checkpointId)
                                   throws Exception

Specified by:

snapshotState in interface org.apache.flink.api.connector.source.SplitEnumerator<KafkaPartitionSplit,KafkaSourceEnumState>

Throws:

Exception

close

public void close()

Specified by:

close in interface AutoCloseable

Specified by:

close in interface org.apache.flink.api.connector.source.SplitEnumerator<KafkaPartitionSplit,KafkaSourceEnumState>