Package org.apache.flink.connector.kafka.source.enumerator.initializer

Interface OffsetsInitializer

All Superinterfaces:
Serializable
All Known Implementing Classes:
NoStoppingOffsetsInitializer
@PublicEvolving public interface OffsetsInitializer extends Serializable
An interface for users to specify the starting / stopping offset of a KafkaPartitionSplit.
See Also:
  • ReaderHandledOffsetsInitializer
  • SpecifiedOffsetsInitializer
  • TimestampOffsetsInitializer

  • Method Details

    • getPartitionOffsets

      Map<org.apache.kafka.common.TopicPartition,Long> getPartitionOffsets(Collection<org.apache.kafka.common.TopicPartition> partitions, OffsetsInitializer.PartitionOffsetsRetriever partitionOffsetsRetriever)
      Get the initial offsets for the given Kafka partitions. These offsets will be used as either starting offsets or stopping offsets of the Kafka partitions.

      If the implementation returns a starting offset which causes OffsetsOutOfRangeException from Kafka. The OffsetResetStrategy provided by the getAutoOffsetResetStrategy() will be used to reset the offset.

      Parameters:
      partitions - the Kafka partitions to get the starting offsets.
      partitionOffsetsRetriever - a helper to retrieve information of the Kafka partitions.
      Returns:
      A mapping from Kafka partition to their offsets to start consuming from.

    • getAutoOffsetResetStrategy

      org.apache.kafka.clients.consumer.OffsetResetStrategy getAutoOffsetResetStrategy()
      Get the auto offset reset strategy in case the initialized offsets falls out of the range.

      The OffsetStrategy is only used when the offset initializer is used to initialize the starting offsets and the starting offsets is out of range.

      Returns:
      An OffsetResetStrategy to use if the initialized offsets are out of the range.

    • committedOffsets

      static OffsetsInitializer committedOffsets()
      Get an OffsetsInitializer which initializes the offsets to the committed offsets. An exception will be thrown at runtime if there is no committed offsets.
      Returns:
      an offset initializer which initialize the offsets to the committed offsets.

    • committedOffsets

      static OffsetsInitializer committedOffsets(org.apache.kafka.clients.consumer.OffsetResetStrategy offsetResetStrategy)
      Get an OffsetsInitializer which initializes the offsets to the committed offsets. Use the given OffsetResetStrategy to initialize the offsets if the committed offsets does not exist.
      Parameters:
      offsetResetStrategy - the offset reset strategy to use when the committed offsets do not exist.
      Returns:
      an OffsetsInitializer which initializes the offsets to the committed offsets.

    • timestamp

      static OffsetsInitializer timestamp(long timestamp)
      Get an OffsetsInitializer which initializes the offsets in each partition so that the initialized offset is the offset of the first record whose record timestamp is greater than or equals the given timestamp (milliseconds).
      Parameters:
      timestamp - the timestamp (milliseconds) to start the consumption.
      Returns:
      an OffsetsInitializer which initializes the offsets based on the given timestamp.
      See Also:
      • Admin.listOffsets(Map)

    • earliest

      static OffsetsInitializer earliest()
      Get an OffsetsInitializer which initializes the offsets to the earliest available offsets of each partition.
      Returns:
      an OffsetsInitializer which initializes the offsets to the earliest available offsets.

    • latest

      static OffsetsInitializer latest()
      Get an OffsetsInitializer which initializes the offsets to the latest offsets of each partition.
      Returns:
      an OffsetsInitializer which initializes the offsets to the latest offsets.

    • offsets

      static OffsetsInitializer offsets(Map<org.apache.kafka.common.TopicPartition,Long> offsets)
      Get an OffsetsInitializer which initializes the offsets to the specified offsets.
      Parameters:
      offsets - the specified offsets for each partition.
      Returns:
      an OffsetsInitializer which initializes the offsets to the specified offsets.

    • offsets

      static OffsetsInitializer offsets(Map<org.apache.kafka.common.TopicPartition,Long> offsets, org.apache.kafka.clients.consumer.OffsetResetStrategy offsetResetStrategy)
      Get an OffsetsInitializer which initializes the offsets to the specified offsets. Use the given OffsetResetStrategy to initialize the offsets in case the specified offset is out of range.
      Parameters:
      offsets - the specified offsets for each partition.
      offsetResetStrategy - the OffsetResetStrategy to use when the specified offset is out of range.
      Returns:
      an OffsetsInitializer which initializes the offsets to the specified offsets.