org.apache.flink.connector.kafka.dynamic.source.enumerator

Class DynamicKafkaSourceEnumerator

java.lang.Object

org.apache.flink.connector.kafka.dynamic.source.enumerator.DynamicKafkaSourceEnumerator

All Implemented Interfaces:

AutoCloseable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.api.connector.source.SplitEnumerator<DynamicKafkaSourceSplit,DynamicKafkaSourceEnumState>

@Internal
public class DynamicKafkaSourceEnumerator
extends Object
implements org.apache.flink.api.connector.source.SplitEnumerator<DynamicKafkaSourceSplit,DynamicKafkaSourceEnumState>

This enumerator manages multiple KafkaSourceEnumerator's, which does not have any synchronization since it assumes single threaded execution.

Constructor Summary

Constructors

Constructor and Description

DynamicKafkaSourceEnumerator(KafkaStreamSubscriber kafkaStreamSubscriber, KafkaMetadataService kafkaMetadataService, org.apache.flink.api.connector.source.SplitEnumeratorContext<DynamicKafkaSourceSplit> enumContext, OffsetsInitializer startingOffsetsInitializer, OffsetsInitializer stoppingOffsetInitializer, Properties properties, org.apache.flink.api.connector.source.Boundedness boundedness, DynamicKafkaSourceEnumState dynamicKafkaSourceEnumState)

Method Summary

Modifier and Type	Method and Description
void	addReader(int subtaskId) NOTE: this happens at startup and failover.
void	addSplitsBack(List<DynamicKafkaSourceSplit> splits, int subtaskId)
void	close()
void	handleSourceEvent(int subtaskId, org.apache.flink.api.connector.source.SourceEvent sourceEvent)
void	handleSplitRequest(int subtaskId, String requesterHostname) Multi cluster Kafka source readers will not request splits.
DynamicKafkaSourceEnumState	snapshotState(long checkpointId) Besides for checkpointing, this method is used in the restart sequence to retain the relevant assigned splits so that there is no reader duplicate split assignment.
void	start() Discover Kafka clusters and initialize sub enumerators.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.connector.source.SplitEnumerator

notifyCheckpointComplete

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted

Constructor Detail

DynamicKafkaSourceEnumerator

public DynamicKafkaSourceEnumerator(KafkaStreamSubscriber kafkaStreamSubscriber,
                                    KafkaMetadataService kafkaMetadataService,
                                    org.apache.flink.api.connector.source.SplitEnumeratorContext<DynamicKafkaSourceSplit> enumContext,
                                    OffsetsInitializer startingOffsetsInitializer,
                                    OffsetsInitializer stoppingOffsetInitializer,
                                    Properties properties,
                                    org.apache.flink.api.connector.source.Boundedness boundedness,
                                    DynamicKafkaSourceEnumState dynamicKafkaSourceEnumState)

Method Detail

start

public void start()

Discover Kafka clusters and initialize sub enumerators. Bypass kafka metadata service discovery if there exists prior state. Exceptions with initializing Kafka source are treated the same as Kafka state and metadata inconsistency.

Specified by:

start in interface org.apache.flink.api.connector.source.SplitEnumerator<DynamicKafkaSourceSplit,DynamicKafkaSourceEnumState>

handleSplitRequest

public void handleSplitRequest(int subtaskId,
                               @Nullable
                               String requesterHostname)

Multi cluster Kafka source readers will not request splits. Splits will be pushed to them, similarly for the sub enumerators.

Specified by:

handleSplitRequest in interface org.apache.flink.api.connector.source.SplitEnumerator<DynamicKafkaSourceSplit,DynamicKafkaSourceEnumState>

addSplitsBack

public void addSplitsBack(List<DynamicKafkaSourceSplit> splits,
                          int subtaskId)

Specified by:

addSplitsBack in interface org.apache.flink.api.connector.source.SplitEnumerator<DynamicKafkaSourceSplit,DynamicKafkaSourceEnumState>

addReader

public void addReader(int subtaskId)

NOTE: this happens at startup and failover.

Specified by:

addReader in interface org.apache.flink.api.connector.source.SplitEnumerator<DynamicKafkaSourceSplit,DynamicKafkaSourceEnumState>

snapshotState

public DynamicKafkaSourceEnumState snapshotState(long checkpointId)
                                          throws Exception

Besides for checkpointing, this method is used in the restart sequence to retain the relevant assigned splits so that there is no reader duplicate split assignment. See createEnumeratorWithAssignedTopicPartitions(String, Set, KafkaSourceEnumState, Properties)}

Specified by:

snapshotState in interface org.apache.flink.api.connector.source.SplitEnumerator<DynamicKafkaSourceSplit,DynamicKafkaSourceEnumState>

Throws:

Exception

handleSourceEvent

public void handleSourceEvent(int subtaskId,
                              org.apache.flink.api.connector.source.SourceEvent sourceEvent)

Specified by:

handleSourceEvent in interface org.apache.flink.api.connector.source.SplitEnumerator<DynamicKafkaSourceSplit,DynamicKafkaSourceEnumState>

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface org.apache.flink.api.connector.source.SplitEnumerator<DynamicKafkaSourceSplit,DynamicKafkaSourceEnumState>

Throws:

IOException