Package com.google.cloud.hadoop.fs.gcs

Class GhfsInstrumentation

java.lang.Object
com.google.cloud.hadoop.fs.gcs.GhfsInstrumentation
All Implemented Interfaces:
Closeable, AutoCloseable, org.apache.hadoop.fs.statistics.DurationTrackerFactory, org.apache.hadoop.fs.statistics.IOStatisticsSource, org.apache.hadoop.metrics2.MetricsSource
public class GhfsInstrumentation extends Object implements Closeable, org.apache.hadoop.metrics2.MetricsSource, org.apache.hadoop.fs.statistics.IOStatisticsSource, org.apache.hadoop.fs.statistics.DurationTrackerFactory
Instrumentation of GCS.

Counters and metrics are generally addressed in code by their name or GhfsStatistic key. There may be some Statistics which do not have an entry here. To avoid attempts to access such counters failing, the operations to increment/query metric values are designed to handle lookup failures.

GoogleHadoopFileSystem StorageStatistics are dynamically derived from the IOStatistics.

  • Constructor Summary

    Constructors
    Constructor
    Description
    GhfsInstrumentation(URI name)
    Construct the instrumentation for a filesystem.

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    close()
     
    protected final org.apache.hadoop.metrics2.lib.MutableCounterLong
    counter(GhfsStatistic op)
    Create a counter in the registry for metrics in GhfsStatistic.
    protected final org.apache.hadoop.metrics2.lib.MutableCounterLong
    counter(String name, String desc)
    Create a counter in the registry.
    protected final void
    duration(GhfsStatistic op)
    Registering a duration adds the success and failure counters.
    protected final org.apache.hadoop.metrics2.lib.MutableGaugeLong
    gauge(GhfsStatistic op)
    Create a gauge in the registry.
    protected final org.apache.hadoop.metrics2.lib.MutableGaugeLong
    gauge(String name, String desc)
    Create a gauge in the registry.
    org.apache.hadoop.fs.statistics.DurationTrackerFactory
    getDurationTrackerFactory()
    Get the duration tracker factory.
    org.apache.hadoop.fs.statistics.impl.IOStatisticsStore
    getIOStatistics()
    Get the instance IO Statistics.
    void
    getMetrics(org.apache.hadoop.metrics2.MetricsCollector metricsCollector, boolean b)
     
    org.apache.hadoop.metrics2.MetricsSystem
    getMetricsSystem()
    Get the metrics system.
    org.apache.hadoop.metrics2.lib.MetricsRegistry
    getRegistry()
    Get the metrics registry.
    void
    incrementCounter(GhfsStatistic op, long count)
    Increments a mutable counter and the matching instance IOStatistics counter.
    org.apache.hadoop.metrics2.lib.MutableMetric
    lookupMetric(String name)
    Look up a metric from both the registered set and the lighter weight stream entries.
    DelegationTokenStatistics
    newDelegationTokenStatistics()
    Create a delegation token statistics instance.
    com.google.cloud.hadoop.fs.gcs.GhfsInputStreamStatistics
    newInputStreamStatistics(org.apache.hadoop.fs.FileSystem.Statistics filesystemStatistics)
    Create a stream input statistics instance.
    com.google.cloud.hadoop.fs.gcs.GhfsOutputStreamStatistics
    newOutputStreamStatistics(org.apache.hadoop.fs.FileSystem.Statistics filesystemStatistics)
    Create a stream output statistics instance.
    org.apache.hadoop.fs.statistics.DurationTracker
    trackDuration(String key, long count)
    The duration tracker updates the metrics with the count and IOStatistics will full duration information.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.hadoop.fs.statistics.DurationTrackerFactory

    trackDuration

  • Constructor Details

    • GhfsInstrumentation

      public GhfsInstrumentation(URI name)
      Construct the instrumentation for a filesystem.
      Parameters:
      name - URI of filesystem.

  • Method Details

    • close

      public void close()
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable

    • getIOStatistics

      public org.apache.hadoop.fs.statistics.impl.IOStatisticsStore getIOStatistics()
      Get the instance IO Statistics.
      Specified by:
      getIOStatistics in interface org.apache.hadoop.fs.statistics.IOStatisticsSource
      Returns:
      statistics.

    • incrementCounter

      public void incrementCounter(GhfsStatistic op, long count)
      Increments a mutable counter and the matching instance IOStatistics counter. No-op if the counter is not defined, or the count == 0.
      Parameters:
      op - operation
      count - increment value

    • getMetricsSystem

      public org.apache.hadoop.metrics2.MetricsSystem getMetricsSystem()
      Get the metrics system.
      Returns:
      metricsSystem

    • counter

      protected final org.apache.hadoop.metrics2.lib.MutableCounterLong counter(String name, String desc)
      Create a counter in the registry.
      Parameters:
      name - counter name
      desc - counter description
      Returns:
      a new counter

    • counter

      protected final org.apache.hadoop.metrics2.lib.MutableCounterLong counter(GhfsStatistic op)
      Create a counter in the registry for metrics in GhfsStatistic.
      Parameters:
      op - statistic to count
      Returns:
      a new counter

    • duration

      protected final void duration(GhfsStatistic op)
      Registering a duration adds the success and failure counters.
      Parameters:
      op - statistic to track

    • gauge

      protected final org.apache.hadoop.metrics2.lib.MutableGaugeLong gauge(String name, String desc)
      Create a gauge in the registry.
      Parameters:
      name - name gauge name
      desc - description
      Returns:
      the gauge

    • gauge

      protected final org.apache.hadoop.metrics2.lib.MutableGaugeLong gauge(GhfsStatistic op)
      Create a gauge in the registry.
      Parameters:
      op - statistic to count
      Returns:
      the gauge

    • getRegistry

      public org.apache.hadoop.metrics2.lib.MetricsRegistry getRegistry()
      Get the metrics registry.
      Returns:
      the registry

    • lookupMetric

      public org.apache.hadoop.metrics2.lib.MutableMetric lookupMetric(String name)
      Look up a metric from both the registered set and the lighter weight stream entries.
      Parameters:
      name - metric name
      Returns:
      the metric or null

    • getDurationTrackerFactory

      public org.apache.hadoop.fs.statistics.DurationTrackerFactory getDurationTrackerFactory()
      Get the duration tracker factory.
      Returns:
      duration tracking for the instrumentation.

    • trackDuration

      public org.apache.hadoop.fs.statistics.DurationTracker trackDuration(String key, long count)
      The duration tracker updates the metrics with the count and IOStatistics will full duration information.
      Specified by:
      trackDuration in interface org.apache.hadoop.fs.statistics.DurationTrackerFactory
      Parameters:
      key - statistic key prefix
      count - #of times to increment the matching counter in this operation.
      Returns:
      a duration tracker.

    • getMetrics

      public void getMetrics(org.apache.hadoop.metrics2.MetricsCollector metricsCollector, boolean b)
      Specified by:
      getMetrics in interface org.apache.hadoop.metrics2.MetricsSource

    • newInputStreamStatistics

      public com.google.cloud.hadoop.fs.gcs.GhfsInputStreamStatistics newInputStreamStatistics(@Nullable org.apache.hadoop.fs.FileSystem.Statistics filesystemStatistics)
      Create a stream input statistics instance.
      Parameters:
      filesystemStatistics - FS Statistics to update in close().
      Returns:
      the new instance

    • newOutputStreamStatistics

      public com.google.cloud.hadoop.fs.gcs.GhfsOutputStreamStatistics newOutputStreamStatistics(org.apache.hadoop.fs.FileSystem.Statistics filesystemStatistics)
      Create a stream output statistics instance.
      Parameters:
      filesystemStatistics - thread-local FS statistics.
      Returns:
      the new instance

    • newDelegationTokenStatistics

      public DelegationTokenStatistics newDelegationTokenStatistics()
      Create a delegation token statistics instance.
      Returns:
      an instance of delegation token statistics