Package org.apache.flink.connector.kafka.source.reader

Class KafkaSourceReader<T>

java.lang.Object

org.apache.flink.connector.base.source.reader.SourceReaderBase<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,T,KafkaPartitionSplit,KafkaPartitionSplitState>

org.apache.flink.connector.base.source.reader.SingleThreadMultiplexSourceReaderBase<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,T,KafkaPartitionSplit,KafkaPartitionSplitState>

org.apache.flink.connector.kafka.source.reader.KafkaSourceReader<T>

All Implemented Interfaces:

AutoCloseable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.api.connector.source.SourceReader<T,KafkaPartitionSplit>

@Internal
public class KafkaSourceReader<T>
extends org.apache.flink.connector.base.source.reader.SingleThreadMultiplexSourceReaderBase<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,T,KafkaPartitionSplit,KafkaPartitionSplitState>

The source reader for Kafka partitions.

Field Summary

Fields inherited from class org.apache.flink.connector.base.source.reader.SourceReaderBase

config, context, eofRecordEvaluator, options, recordEmitter, splitFetcherManager

Constructor Summary

Constructors

Constructor

Description

KafkaSourceReader(org.apache.flink.connector.base.source.reader.synchronization.FutureCompletingBlockingQueue<org.apache.flink.connector.base.source.reader.RecordsWithSplitIds<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>>> elementsQueue, KafkaSourceFetcherManager kafkaSourceFetcherManager, org.apache.flink.connector.base.source.reader.RecordEmitter<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,T,KafkaPartitionSplitState> recordEmitter, org.apache.flink.configuration.Configuration config, org.apache.flink.api.connector.source.SourceReaderContext context, KafkaSourceReaderMetrics kafkaSourceReaderMetrics)

Method Summary

Modifier and Type	Method	Description
protected KafkaPartitionSplitState	initializedState(KafkaPartitionSplit split)
void	notifyCheckpointComplete(long checkpointId)
protected void	onSplitFinished(Map<String,KafkaPartitionSplitState> finishedSplitIds)
List<KafkaPartitionSplit>	snapshotState(long checkpointId)
protected KafkaPartitionSplit	toSplitType(String splitId, KafkaPartitionSplitState splitState)

Methods inherited from class org.apache.flink.connector.base.source.reader.SourceReaderBase

addSplits, close, getNumberOfCurrentlyAssignedSplits, handleSourceEvents, isAvailable, notifyNoMoreSplits, pauseOrResumeSplits, pollNext, start

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted

Constructor Details

KafkaSourceReader

public KafkaSourceReader(org.apache.flink.connector.base.source.reader.synchronization.FutureCompletingBlockingQueue<org.apache.flink.connector.base.source.reader.RecordsWithSplitIds<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>>> elementsQueue,
 KafkaSourceFetcherManager kafkaSourceFetcherManager,
 org.apache.flink.connector.base.source.reader.RecordEmitter<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,T,KafkaPartitionSplitState> recordEmitter,
 org.apache.flink.configuration.Configuration config,
 org.apache.flink.api.connector.source.SourceReaderContext context,
 KafkaSourceReaderMetrics kafkaSourceReaderMetrics)

Method Details

onSplitFinished

protected void onSplitFinished(Map<String,KafkaPartitionSplitState> finishedSplitIds)

Specified by:

onSplitFinished in class org.apache.flink.connector.base.source.reader.SourceReaderBase<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,T,KafkaPartitionSplit,KafkaPartitionSplitState>

snapshotState

public List<KafkaPartitionSplit> snapshotState(long checkpointId)

Specified by:

snapshotState in interface org.apache.flink.api.connector.source.SourceReader<T,KafkaPartitionSplit>

Overrides:

snapshotState in class org.apache.flink.connector.base.source.reader.SourceReaderBase<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,T,KafkaPartitionSplit,KafkaPartitionSplitState>

notifyCheckpointComplete

public void notifyCheckpointComplete(long checkpointId)
                              throws Exception

Throws:

Exception

initializedState

protected KafkaPartitionSplitState initializedState(KafkaPartitionSplit split)

Specified by:

initializedState in class org.apache.flink.connector.base.source.reader.SourceReaderBase<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,T,KafkaPartitionSplit,KafkaPartitionSplitState>

toSplitType

protected KafkaPartitionSplit toSplitType(String splitId,
 KafkaPartitionSplitState splitState)

Specified by:

toSplitType in class org.apache.flink.connector.base.source.reader.SourceReaderBase<org.apache.kafka.clients.consumer.ConsumerRecord<byte[],byte[]>,T,KafkaPartitionSplit,KafkaPartitionSplitState>