Package org.apache.flink.connector.kafka.dynamic.source

Class DynamicKafkaSource<T>

  • Type Parameters:
    T - Record type
    All Implemented Interfaces:
    Serializable, org.apache.flink.api.connector.source.Source<T,​DynamicKafkaSourceSplit,​DynamicKafkaSourceEnumState>, org.apache.flink.api.connector.source.SourceReaderFactory<T,​DynamicKafkaSourceSplit>, org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>
    @Experimental
public class DynamicKafkaSource<T>
extends Object
implements org.apache.flink.api.connector.source.Source<T,​DynamicKafkaSourceSplit,​DynamicKafkaSourceEnumState>, org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>
    Factory class for the DynamicKafkaSource components. FLIP-246: DynamicKafkaSource

    This source's key difference from KafkaSource is that it enables users to read dynamically, which does not require job restart, from streams (topics that belong to one or more clusters). If using KafkaSource, users need to restart the job by deleting the job and reconfiguring the topics and clusters.

    This example shows how to configure a DynamicKafkaSource that emits Integer records: 

    
 DynamicKafkaSource<Integer> dynamicKafkaSource =
                     DynamicKafkaSource.<Integer>builder()
                             .setStreamIds(Collections.singleton("MY_STREAM_ID"))
                             // custom metadata service that resolves `MY_STREAM_ID` to the associated clusters and topics
                             .setKafkaMetadataService(kafkaMetadataService)
                             .setDeserializer(
                                     KafkaRecordDeserializationSchema.valueOnly(
                                             IntegerDeserializer.class))
                             .setStartingOffsets(OffsetsInitializer.earliest())
                             // common properties for all Kafka clusters
                             .setProperties(properties)
                             .build();

    See more configuration options in DynamicKafkaSourceBuilder and DynamicKafkaSourceOptions.

    See Also:
    Serialized Form