Package org.apache.flink.connector.kafka.dynamic.source

Class DynamicKafkaSource<T>

java.lang.Object
org.apache.flink.connector.kafka.dynamic.source.DynamicKafkaSource<T>
Type Parameters:
T - Record type
All Implemented Interfaces:
Serializable, org.apache.flink.api.connector.source.Source<T,DynamicKafkaSourceSplit,DynamicKafkaSourceEnumState>, org.apache.flink.api.connector.source.SourceReaderFactory<T,DynamicKafkaSourceSplit>, org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>
@Experimental public class DynamicKafkaSource<T> extends Object implements org.apache.flink.api.connector.source.Source<T,DynamicKafkaSourceSplit,DynamicKafkaSourceEnumState>, org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>
Factory class for the DynamicKafkaSource components. FLIP-246: DynamicKafkaSource

This source's key difference from KafkaSource is that it enables users to read dynamically, which does not require job restart, from streams (topics that belong to one or more clusters). If using KafkaSource, users need to restart the job by deleting the job and reconfiguring the topics and clusters.

This example shows how to configure a DynamicKafkaSource that emits Integer records: 


 DynamicKafkaSource<Integer> dynamicKafkaSource =
                     DynamicKafkaSource.<Integer>builder()
                             .setStreamIds(Collections.singleton("MY_STREAM_ID"))
                             // custom metadata service that resolves `MY_STREAM_ID` to the associated clusters and topics
                             .setKafkaMetadataService(kafkaMetadataService)
                             .setDeserializer(
                                     KafkaRecordDeserializationSchema.valueOnly(
                                             IntegerDeserializer.class))
                             .setStartingOffsets(OffsetsInitializer.earliest())
                             // common properties for all Kafka clusters
                             .setProperties(properties)
                             .build();

See more configuration options in DynamicKafkaSourceBuilder and DynamicKafkaSourceOptions.

See Also: