Package org.apache.flink.connector.kafka.source.metrics

Class KafkaSourceReaderMetrics

  • @PublicEvolving
public class KafkaSourceReaderMetrics
extends Object
    A collection class for handling metrics in KafkaSourceReader.

    All metrics of Kafka source reader are registered under group "KafkaSourceReader", which is a child group of OperatorMetricGroup. Metrics related to a specific topic partition will be registered in the group "KafkaSourceReader.topic.{topic_name}.partition.{partition_id}".

    For example, current consuming offset of topic "my-topic" and partition 1 will be reported in metric: "{some_parent_groups}.operator.KafkaSourceReader.topic.my-topic.partition.1.currentOffset"

    and number of successful commits will be reported in metric: "{some_parent_groups}.operator.KafkaSourceReader.commitsSucceeded"

    All metrics of Kafka consumer are also registered under group "KafkaSourceReader.KafkaConsumer". For example, Kafka consumer metric "records-consumed-total" can be found at: {some_parent_groups}.operator.KafkaSourceReader.KafkaConsumer.records-consumed-total"

    • Constructor Detail

      • KafkaSourceReaderMetrics

        public KafkaSourceReaderMetrics​(org.apache.flink.metrics.groups.SourceReaderMetricGroup sourceReaderMetricGroup)

    • Method Detail

      • registerKafkaConsumerMetrics

        public void registerKafkaConsumerMetrics​(org.apache.kafka.clients.consumer.KafkaConsumer<?,​?> kafkaConsumer)
        Register metrics of KafkaConsumer in Kafka metric group.
        Parameters:
        kafkaConsumer - Kafka consumer used by partition split reader.

      • registerTopicPartition

        public void registerTopicPartition​(org.apache.kafka.common.TopicPartition tp)
        Register metric groups for the given TopicPartition.
        Parameters:
        tp - Registering topic partition

      • recordCurrentOffset

        public void recordCurrentOffset​(org.apache.kafka.common.TopicPartition tp,
                                long offset)
        Update current consuming offset of the given TopicPartition.
        Parameters:
        tp - Updating topic partition
        offset - Current consuming offset

      • recordCommittedOffset

        public void recordCommittedOffset​(org.apache.kafka.common.TopicPartition tp,
                                  long offset)
        Update the latest committed offset of the given TopicPartition.
        Parameters:
        tp - Updating topic partition
        offset - Committing offset

      • recordSucceededCommit

        public void recordSucceededCommit()
        Mark a successful commit.

      • recordFailedCommit

        public void recordFailedCommit()
        Mark a failure commit.

      • registerNumBytesIn

        public void registerNumBytesIn​(org.apache.kafka.clients.consumer.KafkaConsumer<?,​?> consumer)
        Register MetricNames.IO_NUM_BYTES_IN.
        Parameters:
        consumer - Kafka consumer

      • maybeAddRecordsLagMetric

        public void maybeAddRecordsLagMetric​(org.apache.kafka.clients.consumer.KafkaConsumer<?,​?> consumer,
                                     org.apache.kafka.common.TopicPartition tp)
        Add a partition's records-lag metric to tracking list if this partition never appears before.

        This method also lazily register MetricNames.PENDING_RECORDS in SourceReaderMetricGroup

        Parameters:
        consumer - Kafka consumer
        tp - Topic partition

      • removeRecordsLagMetric

        public void removeRecordsLagMetric​(org.apache.kafka.common.TopicPartition tp)
        Remove a partition's records-lag metric from tracking list.
        Parameters:
        tp - Unassigned topic partition

      • updateNumBytesInCounter

        public void updateNumBytesInCounter()
        Update MetricNames.IO_NUM_BYTES_IN.

        Instead of simply setting OperatorIOMetricGroup.getNumBytesInCounter() to the same value as bytes-consumed-total from Kafka consumer, which will screw TaskIOMetricGroup.getNumBytesInCounter() if chained sources exist, we track the increment of bytes-consumed-total and count it towards the counter.