Package org.apache.druid.query

Interface QueryMetrics<QueryType extends Query<?>>

Type Parameters:
QueryType -
All Known Subinterfaces:
GroupByQueryMetrics, SearchQueryMetrics, TimeseriesQueryMetrics, TopNQueryMetrics
All Known Implementing Classes:
DefaultGroupByQueryMetrics, DefaultQueryMetrics, DefaultSearchQueryMetrics, DefaultTimeseriesQueryMetrics, DefaultTopNQueryMetrics
public interface QueryMetrics<QueryType extends Query<?>>
Abstraction wrapping ServiceMetricEvent.Builder and allowing to control what metrics are actually emitted, what dimensions do they have, etc. Goals of QueryMetrics --------------------- 1. Skipping or partial filtering of particular dimensions or metrics entirely. Implementation could leave the body of the corresponding method empty, or implement random filtering like: public void reportCpuTime(long timeNs) { if (ThreadLocalRandom.current().nextDouble() < 0.1) { super.reportCpuTime(timeNs); } } 2. Ability to add new dimensions and metrics, possibly expensive to compute, or expensive to process (long string values, high cardinality, etc.) and not to affect existing Druid installations, by skipping (see 1.) those dimensions and metrics entirely in the default QueryMetrics implementations. Users who need those expensive dimensions and metrics, could explicitly emit them in their own QueryMetrics. 3. Control over the time unit, in which time metrics are emitted. By default (see DefaultQueryMetrics and it's subclasses) it's milliseconds, but if queries are fast, it could be not precise enough. 4. Control over the dimension and metric names. Here, "control" is provided to the operator of a Druid cluster, who would exercise that control through a site-specific extension adding XxxQueryMetricsFactory impl(s). Types of methods in this interface ---------------------------------- 1. Methods, pulling some dimensions from the query object. These methods are used to populate the metric before the query is run. These methods accept a single `QueryType query` parameter. query(Query) calls all methods of this type, hence pulling all available information from the query object as dimensions. 2. Methods for setting dimensions, which become known in the process of the query execution or after the query is completed. 3. Methods to register metrics to be emitted later in bulk via emit(ServiceEmitter). These methods return this QueryMetrics object back for chaining. Names of these methods start with "report" prefix. Implementors expectations ------------------------- QueryMetrics is expected to be changed often, in every Druid release (including "patch" releases). Users who create their custom implementations of QueryMetrics should be ready to fix the code of their QueryMetrics (implement new methods) when they update Druid. Broken builds of custom extensions, containing custom QueryMetrics is the way to notify users that Druid core "wants" to emit new dimension or metric, and the user handles them manually: if the new dimension or metric is useful and not very expensive to process and store then emit, skip (see above Goals, 1.) otherwise.

Despite this interface is annotated as ExtensionPoint and some of it's methods as PublicApi, it may be changed in breaking ways even in minor releases.

If implementors of custom QueryMetrics don't want to fix builds on every Druid release (e. g. if they want to add a single dimension to emitted events and don't want to alter other dimensions and emitted metrics), they could inherit their custom QueryMetrics from DefaultQueryMetrics or query-specific default implementation class, such as DefaultTopNQueryMetrics. Those classes are guaranteed to stay around and implement new methods, added to the QueryMetrics interface (or a query-specific subinterface). However, there is no 100% guarantee of compatibility, because methods could not only be added to QueryMetrics, existing methods could also be changed or removed.

QueryMetrics is designed for use from a single thread, implementations shouldn't care about thread-safety. Adding new methods to QueryMetrics ---------------------------------- 1. When adding a new method for setting a dimension, which could be pulled from the query object, always make them accept a single `QueryType query` parameter, letting the implementations to do all the work of carving the dimension value out of the query object. 2. When adding a new method for setting a dimension, which becomes known in the process of the query execution or after the query is completed, design it so that as little work as possible is done for preparing arguments for this method, and as much work as possible is done in the implementations of this method, if they decide to actually emit this dimension. 3. When adding a new method for registering metrics, make it to accept the metric value in the smallest reasonable unit (i. e. nanoseconds for time metrics, bytes for metrics of data size, etc.), allowing the implementations of this method to round the value up to more coarse-grained units, if they don't need the maximum precision. Making subinterfaces of QueryMetrics for emitting custom dimensions and/or metrics for specific query types ----------------------------------------------------------------------------------------------------------- If a query type (e. g. SegmentMetadataQuery (it's runners) needs to emit custom dimensions and/or metrics which doesn't make sense for all other query types, the following steps should be executed: 1. Create `interface SegmentMetadataQueryMetrics extends QueryMetrics` (here and below "SegmentMetadata" is the query type) with additional methods (see "Adding new methods" section above). 2. Create `class DefaultSegmentMetadataQueryMetrics implements SegmentMetadataQueryMetrics`. This class should implement extra methods from SegmentMetadataQueryMetrics interfaces with empty bodies, AND DELEGATE ALL OTHER METHODS TO A QueryMetrics OBJECT, provided as a sole parameter in DefaultSegmentMetadataQueryMetrics constructor. NOTE: query(), dataSource(), queryType(), interval(), hasFilters(), duration(), queryId(), sqlQueryId(), and context() methods or any "pre-query-execution-time" methods should either have a empty body or throw exception. 3. Create `interface SegmentMetadataQueryMetricsFactory` with a single method `SegmentMetadataQueryMetrics makeMetrics(SegmentMetadataQuery query);`. 4. Create `class DefaultSegmentMetadataQueryMetricsFactory implements SegmentMetadataQueryMetricsFactory`, which accepts GenericQueryMetricsFactory as injected constructor parameter, and implements makeMetrics() as `return new DefaultSegmentMetadataQueryMetrics(genericQueryMetricsFactory.makeMetrics(query));` 5. Inject and use SegmentMetadataQueryMetricsFactory instead of GenericQueryMetricsFactory in SegmentMetadataQueryQueryToolChest. 6. Establish injection of SegmentMetadataQueryMetricsFactory using config and provider method in QueryToolChestModule (see how it is done in QueryToolChestModule) for existing query types with custom metrics, e. g. SearchQueryMetricsFactory), if the query type belongs to the core druid-processing, e. g. SegmentMetadataQuery. If the query type defined in an extension, you can specify `binder.bind(ScanQueryMetricsFactory.class).to(DefaultScanQueryMetricsFactory.class)` in the extension's Guice module, if the query type is defined in an extension, e. g. ScanQuery. Or establish similar configuration, as for the core query types. This complex procedure is needed to ensure custom GenericQueryMetricsFactory specified by users still works for the query type when query type decides to create their custom QueryMetrics subclass. TopNQueryMetrics, GroupByQueryMetrics, and TimeseriesQueryMetrics are implemented differently, because they are introduced at the same time as the whole QueryMetrics abstraction and their default implementations have to actually emit more dimensions than the default generic QueryMetrics. So those subinterfaces shouldn't be taken as direct examples for following the plan specified above. Refer SearchQueryMetricsFactory as an implementation example of this procedure.

  • Method Details

    • query

      void query(QueryType query)
      Pulls all information from the query object into dimensions of future metrics.

    • dataSource

      void dataSource(QueryType query)
      Sets Query.getDataSource() of the given query as dimension.

    • queryType

      void queryType(QueryType query)
      Sets Query.getType() of the given query as dimension.

    • interval

      void interval(QueryType query)
      Sets Query.getIntervals() of the given query as dimension.

    • hasFilters

      void hasFilters(QueryType query)
      Sets Query.hasFilters() of the given query as dimension.

    • duration

      void duration(QueryType query)
      Sets Query.getDuration() of the given query as dimension.

    • queryId

      void queryId(QueryType query)
      Sets Query.getId() of the given query as dimension.

    • queryId

      void queryId(String queryId)
      Sets id of the given query as dimension.

    • subQueryId

      void subQueryId(QueryType query)
      Sets Query.getSubQueryId() of the given query as dimension.

    • sqlQueryId

      void sqlQueryId(QueryType query)
      Sets Query.getSqlQueryId() of the given query as dimension

    • sqlQueryId

      void sqlQueryId(String sqlQueryId)
      Sets sqlQueryId as a dimension

    • context

      void context(QueryType query)
      Sets Query.getContext() of the given query as dimension.

    • server

      void server(String host)

    • remoteAddress

      void remoteAddress(String remoteAddress)

    • status

      void status(String status)

    • success

      void success(boolean success)

    • statusCode

      void statusCode(int code)
      Sets the query status code See DruidMetrics.computeStatusCode(java.lang.Throwable).

    • segment

      void segment(String segmentIdentifier)

    • projection

      void projection(String projection)
      If a projection was used during segment processing, set its name as the projection dimension

    • preFilters

      @Deprecated default void preFilters(List<Filter> preFilters)
      Deprecated.
      use filterBundle(FilterBundle.BundleInfo) instead to collect details about filters which were used to construct BitmapOffset or BitmapVectorOffset. This method will be removed in a future Druid release

    • postFilters

      @Deprecated default void postFilters(List<Filter> postFilters)
      Deprecated.
      use filterBundle(FilterBundle.BundleInfo) instead to collect details about filters which were used as value matchers for FilteredOffset or FilteredVectorOffset This method will be removed in a future Druid release

    • filterBundle

      default void filterBundle(FilterBundle.BundleInfo bundleInfo)

    • identity

      void identity(String identity)
      Sets identity of the requester for a query. See AuthenticationResult.

    • vectorized

      void vectorized(boolean vectorized)
      Sets whether or not a segment scan has been vectorized. Generally expected to only be attached to segment-level metrics, since at whole-query level we might have a mix of vectorized and non-vectorized segment scans.

    • parallelMergeParallelism

      void parallelMergeParallelism(int parallelism)
      Sets broker merge parallelism, if parallel merges are enabled. This will only appear in broker level metrics. This value is identical to the reportParallelMergeParallelism(int) metric value, but optionally also available as a dimension.

    • makeBitmapResultFactory

      BitmapResultFactory<?> makeBitmapResultFactory(BitmapFactory factory)
      Creates a BitmapResultFactory which may record some information along bitmap construction from filterBundle(FilterBundle.BundleInfo). The returned BitmapResultFactory may add some dimensions to this QueryMetrics from it's BitmapResultFactory.toImmutableBitmap(Object) method. See BitmapResultFactory Javadoc for more information.

    • reportQueryTime

      QueryMetrics<QueryType> reportQueryTime(long timeNs)
      Registers "query time" metric. Measures the time between a Jetty thread starting to handle a query, and the response being fully written to the response output stream. Does not include time spent waiting in a queue before the query runs.

    • reportQueryBytes

      QueryMetrics<QueryType> reportQueryBytes(long byteCount)
      Registers "query bytes" metric. Measures the total number of bytes written by the query server thread to the response output stream. Emitted once per query.

    • reportQueriedSegmentCount

      QueryMetrics<QueryType> reportQueriedSegmentCount(long segmentCount)
      Registers "segments queried count" metric.

    • reportWaitTime

      QueryMetrics<QueryType> reportWaitTime(long timeNs)
      Registers "wait time" metric. Measures the total time segment-processing runnables spent waiting for execution in the processing thread pool. Emitted once per segment.

    • reportSegmentTime

      QueryMetrics<QueryType> reportSegmentTime(long timeNs)
      Registers "segment time" metric. Measures the total wall-clock time spent operating on segments in processing threads. Emitted once per segment.

    • reportSegmentAndCacheTime

      QueryMetrics<QueryType> reportSegmentAndCacheTime(long timeNs)
      Registers "segmentAndCache time" metric. Measures the total wall-clock time spent in processing threads, either operating on segments or retrieving items from cache. Emitted once per segment.

    • reportSegmentOnDemandLoadTime

      QueryMetrics<QueryType> reportSegmentOnDemandLoadTime(long timeNs)
      Registers the apparent time spent loading segments on demand, before queing up for processing. This measurement is wall-clock time to when the last segment is finished loading and ready for processing.

      Emitted once per query

    • reportSegmentOnDemandLoadTimeAvg

      QueryMetrics<QueryType> reportSegmentOnDemandLoadTimeAvg(long timeNs)
      Registers the average time spent loading segments on demand across threads.

      Emitted once per query

    • reportSegmentOnDemandLoadWaitTimeMax

      QueryMetrics<QueryType> reportSegmentOnDemandLoadWaitTimeMax(long timeNs)
      Registers the maximum time spent loading segments on demand across all load threads.

      Emitted once per query

    • reportSegmentOnDemandLoadWaitTimeAvg

      QueryMetrics<QueryType> reportSegmentOnDemandLoadWaitTimeAvg(long timeNs)
      Registers the average time spent waiting for a thread to start loading segments on demand across threads.

      Emitted once per query

    • reportSegmentOnDemandLoadTimeMax

      QueryMetrics<QueryType> reportSegmentOnDemandLoadTimeMax(long timeNs)
      Registers the maximum time spent waiting for a thread to start loading segments on demand across all load threads.

      Emitted once per query

    • reportSegmentOnDemandLoadBytes

      QueryMetrics<QueryType> reportSegmentOnDemandLoadBytes(long byteCount)
      Registers the total number of bytes added to the cache when loading segments on demand, summing the sizes loaded by individual segement load threads.

      Emitted once per query

    • reportSegmentOnDemandLoadCount

      QueryMetrics<QueryType> reportSegmentOnDemandLoadCount(long count)
      Registers the total numer of segments loaded on demand.

      Emitted once per query

    • reportResultCachePoll

      QueryMetrics<QueryType> reportResultCachePoll(boolean hit)
      Emits iff a given query polled the result-level cache and the success of that operation.

    • reportCpuTime

      QueryMetrics<QueryType> reportCpuTime(long timeNs)
      Registers "cpu time" metric.

    • reportNodeTimeToFirstByte

      QueryMetrics<QueryType> reportNodeTimeToFirstByte(long timeNs)
      Registers "time to first byte" metric.

    • reportBackPressureTime

      QueryMetrics<QueryType> reportBackPressureTime(long timeNs)
      Registers "time that channel is unreadable (backpressure)" metric.

    • reportNodeTime

      QueryMetrics<QueryType> reportNodeTime(long timeNs)
      Registers "node time" metric.

    • reportNodeBytes

      QueryMetrics<QueryType> reportNodeBytes(long byteCount)
      Registers "node bytes" metric.

    • reportBitmapConstructionTime

      QueryMetrics<QueryType> reportBitmapConstructionTime(long timeNs)
      Reports the time spent constructing bitmap from filterBundle(FilterBundle.BundleInfo) of the query. Not reported, if there are no indexes.

    • reportSegmentRows

      QueryMetrics<QueryType> reportSegmentRows(long numRows)
      Reports the total number of rows in the processed segment.

    • reportPreFilteredRows

      QueryMetrics<QueryType> reportPreFilteredRows(long numRows)
      Reports the number of rows to scan in the segment after applying filterBundle(FilterBundle.BundleInfo). If the are no indexes, this metric is equal to reportSegmentRows(long).

    • reportParallelMergeParallelism

      QueryMetrics<QueryType> reportParallelMergeParallelism(int parallelism)
      Reports number of parallel tasks the broker used to process the query during parallel merge. This value is identical to the parallelMergeParallelism(int) dimension value, but optionally also available as a metric.

    • reportParallelMergeInputSequences

      QueryMetrics<QueryType> reportParallelMergeInputSequences(long numSequences)
      Reports total number of input sequences processed by the broker during parallel merge.

    • reportParallelMergeInputRows

      QueryMetrics<QueryType> reportParallelMergeInputRows(long numRows)
      Reports total number of input rows processed by the broker during parallel merge.

    • reportParallelMergeOutputRows

      QueryMetrics<QueryType> reportParallelMergeOutputRows(long numRows)
      Reports broker total number of output rows after merging and combining input sequences (should be less than or equal to the value supplied to reportParallelMergeInputRows(long).

    • reportParallelMergeTaskCount

      QueryMetrics<QueryType> reportParallelMergeTaskCount(long numTasks)
      Reports broker total number of fork join pool tasks required to complete query

    • reportParallelMergeTotalCpuTime

      QueryMetrics<QueryType> reportParallelMergeTotalCpuTime(long timeNs)
      Reports broker total CPU time in nanoseconds where fork join merge combine tasks were doing work

    • reportParallelMergeTotalTime

      QueryMetrics<QueryType> reportParallelMergeTotalTime(long timeNs)
      Reports broker total "wall" time in nanoseconds from parallel merge start sequence creation to total consumption.

    • reportParallelMergeFastestPartitionTime

      QueryMetrics<QueryType> reportParallelMergeFastestPartitionTime(long timeNs)
      Reports broker "wall" time in nanoseconds for the fastest parallel merge sequence partition to be 'initialized', where 'initialized' is time to the first result batch is populated from data servers and merging can begin. Similar to query 'time to first byte' metrics, except is a composite of the whole group of data servers which are present in the merge partition, which all must supply an initial result batch before merging can actually begin.

    • reportParallelMergeSlowestPartitionTime

      QueryMetrics<QueryType> reportParallelMergeSlowestPartitionTime(long timeNs)
      Reports broker "wall" time in nanoseconds for the slowest parallel merge sequence partition to be 'initialized', where 'initialized' is time to the first result batch is populated from data servers and merging can begin. Similar to query 'time to first byte' metrics, except is a composite of the whole group of data servers which are present in the merge partition, which all must supply an initial result batch before merging can actually begin.

    • emit

      void emit(ServiceEmitter emitter)
      Emits all metrics, registered since the last emit() call on this QueryMetrics object.