Package org.apache.flink.connector.kafka.source.enumerator.initializer

Interface OffsetsInitializer

  • All Superinterfaces:
    Serializable
    All Known Implementing Classes:
    NoStoppingOffsetsInitializer
    @PublicEvolving
public interface OffsetsInitializer
extends Serializable
    An interface for users to specify the starting / stopping offset of a KafkaPartitionSplit.
    See Also:
    ReaderHandledOffsetsInitializer, SpecifiedOffsetsInitializer, TimestampOffsetsInitializer

    • Method Detail

      • getPartitionOffsets

        Map<org.apache.kafka.common.TopicPartition,​Long> getPartitionOffsets​(Collection<org.apache.kafka.common.TopicPartition> partitions,
                                                                           OffsetsInitializer.PartitionOffsetsRetriever partitionOffsetsRetriever)
        Get the initial offsets for the given Kafka partitions. These offsets will be used as either starting offsets or stopping offsets of the Kafka partitions.

        If the implementation returns a starting offset which causes OffsetsOutOfRangeException from Kafka. The OffsetResetStrategy provided by the getAutoOffsetResetStrategy() will be used to reset the offset.

        Parameters:
        partitions - the Kafka partitions to get the starting offsets.
        partitionOffsetsRetriever - a helper to retrieve information of the Kafka partitions.
        Returns:
        A mapping from Kafka partition to their offsets to start consuming from.

      • getAutoOffsetResetStrategy

        org.apache.kafka.clients.consumer.OffsetResetStrategy getAutoOffsetResetStrategy()
        Get the auto offset reset strategy in case the initialized offsets falls out of the range.

        The OffsetStrategy is only used when the offset initializer is used to initialize the starting offsets and the starting offsets is out of range.

        Returns:
        An OffsetResetStrategy to use if the initialized offsets are out of the range.

      • committedOffsets

        static OffsetsInitializer committedOffsets()
        Get an OffsetsInitializer which initializes the offsets to the committed offsets. An exception will be thrown at runtime if there is no committed offsets.
        Returns:
        an offset initializer which initialize the offsets to the committed offsets.

      • committedOffsets

        static OffsetsInitializer committedOffsets​(org.apache.kafka.clients.consumer.OffsetResetStrategy offsetResetStrategy)
        Get an OffsetsInitializer which initializes the offsets to the committed offsets. Use the given OffsetResetStrategy to initialize the offsets if the committed offsets does not exist.
        Parameters:
        offsetResetStrategy - the offset reset strategy to use when the committed offsets do not exist.
        Returns:
        an OffsetsInitializer which initializes the offsets to the committed offsets.

      • timestamp

        static OffsetsInitializer timestamp​(long timestamp)
        Get an OffsetsInitializer which initializes the offsets in each partition so that the initialized offset is the offset of the first record whose record timestamp is greater than or equals the given timestamp (milliseconds).
        Parameters:
        timestamp - the timestamp (milliseconds) to start the consumption.
        Returns:
        an OffsetsInitializer which initializes the offsets based on the given timestamp.
        See Also:
        Admin.listOffsets(Map)

      • offsets

        static OffsetsInitializer offsets​(Map<org.apache.kafka.common.TopicPartition,​Long> offsets)
        Get an OffsetsInitializer which initializes the offsets to the specified offsets.
        Parameters:
        offsets - the specified offsets for each partition.
        Returns:
        an OffsetsInitializer which initializes the offsets to the specified offsets.

      • offsets

        static OffsetsInitializer offsets​(Map<org.apache.kafka.common.TopicPartition,​Long> offsets,
                                  org.apache.kafka.clients.consumer.OffsetResetStrategy offsetResetStrategy)
        Get an OffsetsInitializer which initializes the offsets to the specified offsets. Use the given OffsetResetStrategy to initialize the offsets in case the specified offset is out of range.
        Parameters:
        offsets - the specified offsets for each partition.
        offsetResetStrategy - the OffsetResetStrategy to use when the specified offset is out of range.
        Returns:
        an OffsetsInitializer which initializes the offsets to the specified offsets.