org.apache.flink.connector.kafka.dynamic.source.reader

Class DynamicKafkaSourceReader<T>

java.lang.Object

org.apache.flink.connector.kafka.dynamic.source.reader.DynamicKafkaSourceReader<T>

All Implemented Interfaces:

AutoCloseable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.api.connector.source.SourceReader<T,DynamicKafkaSourceSplit>

@Internal
public class DynamicKafkaSourceReader<T>
extends Object
implements org.apache.flink.api.connector.source.SourceReader<T,DynamicKafkaSourceSplit>

Manages state about underlying KafkaSourceReader to collect records and commit offsets from multiple Kafka clusters. This reader also handles changes to Kafka topology by reacting to restart sequence initiated by the enumerator and suspending inconsistent sub readers.

First, in the restart sequence, we will receive the MetadataUpdateEvent from the enumerator, stop all KafkaSourceReaders, and retain the relevant splits. Second, enumerator will send all new splits that readers should work on (old splits will not be sent again).

Constructor Summary

Constructors

Constructor and Description
DynamicKafkaSourceReader(org.apache.flink.api.connector.source.SourceReaderContext readerContext, KafkaRecordDeserializationSchema<T> deserializationSchema, Properties properties)

Method Summary

Modifier and Type	Method and Description
void	addSplits(List<DynamicKafkaSourceSplit> splits)
void	close()
org.apache.flink.streaming.runtime.io.MultipleFuturesAvailabilityHelper	getAvailabilityHelper()
void	handleSourceEvents(org.apache.flink.api.connector.source.SourceEvent sourceEvent) Duplicate source events are handled with idempotency.
boolean	isActivelyConsumingSplits()
CompletableFuture<Void>	isAvailable()
void	notifyCheckpointComplete(long checkpointId)
void	notifyNoMoreSplits()
org.apache.flink.core.io.InputStatus	pollNext(org.apache.flink.api.connector.source.ReaderOutput<T> readerOutput)
List<DynamicKafkaSourceSplit>	snapshotState(long checkpointId)
void	start() This is invoked first only in reader startup without state.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.connector.source.SourceReader

pauseOrResumeSplits

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted

Constructor Detail

DynamicKafkaSourceReader

public DynamicKafkaSourceReader(org.apache.flink.api.connector.source.SourceReaderContext readerContext,
                                KafkaRecordDeserializationSchema<T> deserializationSchema,
                                Properties properties)

Method Detail

start

public void start()

This is invoked first only in reader startup without state. In stateful startup, splits are added before this method is invoked.

Specified by:

start in interface org.apache.flink.api.connector.source.SourceReader<T,DynamicKafkaSourceSplit>

pollNext

public org.apache.flink.core.io.InputStatus pollNext(org.apache.flink.api.connector.source.ReaderOutput<T> readerOutput)
                                              throws Exception

Specified by:

pollNext in interface org.apache.flink.api.connector.source.SourceReader<T,DynamicKafkaSourceSplit>

Throws:

Exception

addSplits

public void addSplits(List<DynamicKafkaSourceSplit> splits)

Specified by:

addSplits in interface org.apache.flink.api.connector.source.SourceReader<T,DynamicKafkaSourceSplit>

handleSourceEvents

public void handleSourceEvents(org.apache.flink.api.connector.source.SourceEvent sourceEvent)

Duplicate source events are handled with idempotency. No metadata change means we simply skip the restart logic.

Specified by:

handleSourceEvents in interface org.apache.flink.api.connector.source.SourceReader<T,DynamicKafkaSourceSplit>

snapshotState

public List<DynamicKafkaSourceSplit> snapshotState(long checkpointId)

Specified by:

snapshotState in interface org.apache.flink.api.connector.source.SourceReader<T,DynamicKafkaSourceSplit>

isAvailable

public CompletableFuture<Void> isAvailable()

Specified by:

isAvailable in interface org.apache.flink.api.connector.source.SourceReader<T,DynamicKafkaSourceSplit>

notifyNoMoreSplits

public void notifyNoMoreSplits()

Specified by:

notifyNoMoreSplits in interface org.apache.flink.api.connector.source.SourceReader<T,DynamicKafkaSourceSplit>

notifyCheckpointComplete

public void notifyCheckpointComplete(long checkpointId)
                              throws Exception

Specified by:

notifyCheckpointComplete in interface org.apache.flink.api.common.state.CheckpointListener

Specified by:

notifyCheckpointComplete in interface org.apache.flink.api.connector.source.SourceReader<T,DynamicKafkaSourceSplit>

Throws:

Exception

close

public void close()
           throws Exception

Specified by:

close in interface AutoCloseable

Throws:

Exception

getAvailabilityHelper

@VisibleForTesting
public org.apache.flink.streaming.runtime.io.MultipleFuturesAvailabilityHelper getAvailabilityHelper()

isActivelyConsumingSplits

@VisibleForTesting
public boolean isActivelyConsumingSplits()