Package org.apache.flink.connector.kafka.source.metrics

Class KafkaSourceReaderMetrics

java.lang.Object
org.apache.flink.connector.kafka.source.metrics.KafkaSourceReaderMetrics
@PublicEvolving public class KafkaSourceReaderMetrics extends Object
A collection class for handling metrics in KafkaSourceReader.

All metrics of Kafka source reader are registered under group "KafkaSourceReader", which is a child group of OperatorMetricGroup. Metrics related to a specific topic partition will be registered in the group "KafkaSourceReader.topic.{topic_name}.partition.{partition_id}".

For example, current consuming offset of topic "my-topic" and partition 1 will be reported in metric: "{some_parent_groups}.operator.KafkaSourceReader.topic.my-topic.partition.1.currentOffset"

and number of successful commits will be reported in metric: "{some_parent_groups}.operator.KafkaSourceReader.commitsSucceeded"

All metrics of Kafka consumer are also registered under group "KafkaSourceReader.KafkaConsumer". For example, Kafka consumer metric "records-consumed-total" can be found at: {some_parent_groups}.operator.KafkaSourceReader.KafkaConsumer.records-consumed-total"

  • Field Details

  • Constructor Details

    • KafkaSourceReaderMetrics

      public KafkaSourceReaderMetrics(org.apache.flink.metrics.groups.SourceReaderMetricGroup sourceReaderMetricGroup)

  • Method Details

    • registerKafkaConsumerMetrics

      public void registerKafkaConsumerMetrics(org.apache.kafka.clients.consumer.KafkaConsumer<?,?> kafkaConsumer)
      Register metrics of KafkaConsumer in Kafka metric group.
      Parameters:
      kafkaConsumer - Kafka consumer used by partition split reader.

    • registerTopicPartition

      public void registerTopicPartition(org.apache.kafka.common.TopicPartition tp)
      Register metric groups for the given TopicPartition.
      Parameters:
      tp - Registering topic partition

    • recordCurrentOffset

      public void recordCurrentOffset(org.apache.kafka.common.TopicPartition tp, long offset)
      Update current consuming offset of the given TopicPartition.
      Parameters:
      tp - Updating topic partition
      offset - Current consuming offset

    • recordCommittedOffset

      public void recordCommittedOffset(org.apache.kafka.common.TopicPartition tp, long offset)
      Update the latest committed offset of the given TopicPartition.
      Parameters:
      tp - Updating topic partition
      offset - Committing offset

    • recordSucceededCommit

      public void recordSucceededCommit()
      Mark a successful commit.

    • recordFailedCommit

      public void recordFailedCommit()
      Mark a failure commit.

    • registerNumBytesIn

      public void registerNumBytesIn(org.apache.kafka.clients.consumer.KafkaConsumer<?,?> consumer)
      Register MetricNames.IO_NUM_BYTES_IN.
      Parameters:
      consumer - Kafka consumer

    • maybeAddRecordsLagMetric

      public void maybeAddRecordsLagMetric(org.apache.kafka.clients.consumer.KafkaConsumer<?,?> consumer, org.apache.kafka.common.TopicPartition tp)
      Add a partition's records-lag metric to tracking list if this partition never appears before.

      This method also lazily register MetricNames.PENDING_RECORDS in SourceReaderMetricGroup

      Parameters:
      consumer - Kafka consumer
      tp - Topic partition

    • removeRecordsLagMetric

      public void removeRecordsLagMetric(org.apache.kafka.common.TopicPartition tp)
      Remove a partition's records-lag metric from tracking list.
      Parameters:
      tp - Unassigned topic partition

    • updateNumBytesInCounter

      public void updateNumBytesInCounter()
      Update MetricNames.IO_NUM_BYTES_IN.

      Instead of simply setting OperatorIOMetricGroup.getNumBytesInCounter() to the same value as bytes-consumed-total from Kafka consumer, which will screw TaskIOMetricGroup.getNumBytesInCounter() if chained sources exist, we track the increment of bytes-consumed-total and count it towards the counter.