Package org.apache.druid.query.aggregation

Class AggregatorFactory

java.lang.Object
org.apache.druid.query.aggregation.AggregatorFactory
All Implemented Interfaces:
Cacheable
Direct Known Subclasses:
CardinalityAggregatorFactory, CountAggregatorFactory, DoubleAnyAggregatorFactory, DoubleFirstAggregatorFactory, DoubleLastAggregatorFactory, DoubleMeanAggregatorFactory, ExpressionLambdaAggregatorFactory, FilteredAggregatorFactory, FloatAnyAggregatorFactory, FloatFirstAggregatorFactory, FloatLastAggregatorFactory, GroupingAggregatorFactory, HistogramAggregatorFactory, HyperUniquesAggregatorFactory, JavaScriptAggregatorFactory, LongAnyAggregatorFactory, LongFirstAggregatorFactory, LongLastAggregatorFactory, NullableNumericAggregatorFactory, SingleValueAggregatorFactory, StringAnyAggregatorFactory, StringFirstAggregatorFactory, StringLastAggregatorFactory, SuppressedAggregatorFactory
public abstract class AggregatorFactory extends Object implements Cacheable
AggregatorFactory is a strategy (in the terms of Design Patterns) that represents column aggregation, e.g. min, max, sum of metric columns, or cardinality of dimension columns (see CardinalityAggregatorFactory). Implementations of AggregatorFactory which need to Support Nullable Aggregations are encouraged to extend NullableNumericAggregatorFactory. Implementations are also expected to correctly handle single/multi value string type columns as it makes sense for them e.g. doubleSum aggregator tries to parse the string value as double and assumes it to be zero if parsing fails. If it is a multi value column then each individual value should be taken into account for aggregation e.g. if a row had value ["1","1","1"], doubleSum aggregation would take each of them and sum them to 3.

  • Constructor Details

    • AggregatorFactory

      public AggregatorFactory()

  • Method Details

    • factorize

      public abstract Aggregator factorize(ColumnSelectorFactory metricFactory)

    • factorizeBuffered

      public abstract BufferAggregator factorizeBuffered(ColumnSelectorFactory metricFactory)

    • factorizeVector

      public VectorAggregator factorizeVector(VectorColumnSelectorFactory selectorFactory)
      Create a VectorAggregator based on the provided column selector factory. Will throw an exception if this aggregation class does not support vectorization: check "canVectorize" first.

    • factorizeWithSize

      public AggregatorAndSize factorizeWithSize(ColumnSelectorFactory metricFactory)
      Creates an Aggregator based on the provided column selector factory. The returned value is a holder object which contains both the aggregator and its initial size in bytes. The callers can then invoke Aggregator.aggregateWithSize() to perform aggregation and get back the incremental memory required in each aggregate call. Combined with the initial size, this gives the total on-heap memory required by the aggregator.

      This method must include JVM object overheads in the estimated size and must ensure not to underestimate required memory as that might lead to OOM errors.

      This flow does not require invoking guessAggregatorHeapFootprint(long) which tends to over-estimate the required memory.

      Returns:
      AggregatorAndSize which contains the actual aggregator and its initial size.

    • canVectorize

      public boolean canVectorize(ColumnInspector columnInspector)
      Returns whether or not this aggregation class supports vectorization. The default implementation returns false.

    • getComparator

      public abstract Comparator getComparator()

    • combine

      @Nullable public abstract Object combine(@Nullable Object lhs, @Nullable Object rhs)
      A method that knows how to combine the outputs of Aggregator.get() produced via factorize(org.apache.druid.segment.ColumnSelectorFactory) or BufferAggregator.get(java.nio.ByteBuffer, int) produced via factorizeBuffered(org.apache.druid.segment.ColumnSelectorFactory). Note, even though this method is called "combine", this method's contract *does* allow for mutation of the input objects. Thus, any use of lhs or rhs after calling this method is highly discouraged.
      Parameters:
      lhs - The left hand side of the combine
      rhs - The right hand side of the combine
      Returns:
      an object representing the combination of lhs and rhs, this can be a new object or a mutation of the inputs

    • makeAggregateCombiner

      public AggregateCombiner makeAggregateCombiner()
      Creates an AggregateCombiner to fold rollup aggregation results from serveral "rows" of different indexes during index merging. AggregateCombiner implements the same logic as combine(java.lang.Object, java.lang.Object), with the difference that it uses ColumnValueSelector and it's subinterfaces to get inputs and implements ColumnValueSelector to provide output.
      See Also:

    • makeNullableAggregateCombiner

      public AggregateCombiner makeNullableAggregateCombiner()
      Creates an AggregateCombiner which supports nullability. Implementations of AggregatorFactory which need to Support Nullable Aggregations are encouraged to extend NullableNumericAggregatorFactory instead of overriding this method. Default implementation calls makeAggregateCombiner() for backwards compatibility.
      See Also:

    • getCombiningFactory

      public abstract AggregatorFactory getCombiningFactory()
      Returns an AggregatorFactory that can be used to combine the output of aggregators from this factory. It is used when we know we have some values that were produced with this aggregator factory, and want to do some additional combining of them. This happens, for example, when merging query results from two different segments, or two different servers. For simple aggregators, the combining factory may be computed by simply creating a new factory that is the same as the current, except with its input column renamed to the same as the output column. For example, this aggregator: {"type": "longSum", "fieldName": "foo", "name": "bar"} Would become: {"type": "longSum", "fieldName": "bar", "name": "bar"} Sometimes, the type or other parameters of the combining aggregator will be different from the original aggregator. For example, the CountAggregatorFactory getCombiningFactory method will return a LongSumAggregatorFactory, because counts are combined by summing. No matter what, `foo.getCombiningFactory()` and `foo.getCombiningFactory().getCombiningFactory()` should return the same result.
      Returns:
      a new Factory that can be used for operations on top of data output from the current factory.

    • getMergingFactory

      public AggregatorFactory getMergingFactory(AggregatorFactory other) throws AggregatorFactoryNotMergeableException
      Returns an AggregatorFactory that can be used to combine the output of aggregators from this factory and another factory. It is used when we have some values produced by this aggregator factory, and some values produced by the "other" aggregator factory, and we want to do some additional combining of them. This happens, for example, when compacting two segments together that both have a metric column with the same name. (Even though the name of the column is the same, the aggregator factory used to create it may be different from segment to segment.) This method may throw AggregatorFactoryNotMergeableException, meaning that "this" and "other" are not compatible and values from one cannot sensibly be combined with values from the other.
      Returns:
      a new Factory that can be used for merging the output of aggregators from this factory and other.
      Throws:
      AggregatorFactoryNotMergeableException
      See Also:

    • getRequiredColumns

      @Deprecated public List<AggregatorFactory> getRequiredColumns()
      Deprecated.
      This was previously used by group-by v1 and will be removed in a future release

    • deserialize

      public abstract Object deserialize(Object object)
      A method that knows how to "deserialize" the object from whatever form it might have been put into in order to transfer via JSON.
      Parameters:
      object - the object to deserialize
      Returns:
      the deserialized object

    • finalizeComputation

      @Nullable public abstract Object finalizeComputation(@Nullable Object object)
      "Finalizes" the computation of an object. Primarily useful for complex types that have a different mergeable intermediate format than their final resultant output.
      Parameters:
      object - the object to be finalized
      Returns:
      the finalized value that should be returned for the initial query

    • getName

      public abstract String getName()
      Returns:
      output name of the aggregator column.

    • requiredFields

      public abstract List<String> requiredFields()
      Get a list of fields that aggregators built by this factory will need to read.

    • getIntermediateType

      public ColumnType getIntermediateType()
      Get the "intermediate" ColumnType for this aggregator. This is the same as the type returned by deserialize(java.lang.Object) and the type accepted by combine(java.lang.Object, java.lang.Object). However, it is *not* necessarily the same type returned by finalizeComputation(java.lang.Object). Refer to the ColumnType javadocs for details on the implications of choosing a type.

    • getResultType

      public ColumnType getResultType()
      Get the ColumnType for the final form of this aggregator, i.e. the type of the value returned by finalizeComputation(java.lang.Object). This may be the same as or different than the types expected in deserialize(java.lang.Object) and combine(java.lang.Object, java.lang.Object). Refer to the ColumnType javadocs for details on the implications of choosing a type.

    • getType

      @Deprecated public ValueType getType()
      Deprecated.
      This method is deprecated and will be removed soon. Use getIntermediateType() instead. Do not call this method, it will likely produce incorrect results, it exists for backwards compatibility.

    • getFinalizedType

      @Deprecated public ValueType getFinalizedType()
      Deprecated.
      This method is deprecated and will be removed soon. Use getResultType() instead. Do not call this method, it will likely produce incorrect results, it exists for backwards compatibility.

    • getComplexTypeName

      @Nullable @Deprecated public String getComplexTypeName()
      Deprecated.
      This method is deprecated and will be removed soon. Use getIntermediateType() instead. Do not call this method, it will likely produce incorrect results, it exists for backwards compatibility.

    • getMaxIntermediateSize

      public abstract int getMaxIntermediateSize()
      Returns the maximum size that this aggregator will require in bytes for intermediate storage of results.
      Returns:
      the maximum number of bytes that an aggregator of this type will require for intermediate result storage.

    • getMaxIntermediateSizeWithNulls

      public int getMaxIntermediateSizeWithNulls()
      Returns the maximum size that this aggregator will require in bytes for intermediate storage of results. Implementations of AggregatorFactory which need to Support Nullable Aggregations are encouraged to extend NullableNumericAggregatorFactory instead of overriding this method. Default implementation calls makeAggregateCombiner() for backwards compatibility.
      Returns:
      the maximum number of bytes that an aggregator of this type will require for intermediate result storage.

    • guessAggregatorHeapFootprint

      public int guessAggregatorHeapFootprint(long rows)
      Returns a best guess as to how much memory the on-heap Aggregator returned by factorize(org.apache.druid.segment.ColumnSelectorFactory) will require when a certain number of rows have been aggregated into it. The main user of this method is OnheapIncrementalIndex, which uses it to determine when to persist the current in-memory data to disk. Important note for callers! In nearly all cases, callers that wish to constrain memory would be better off using factorizeBuffered(org.apache.druid.segment.ColumnSelectorFactory) or factorizeVector(org.apache.druid.segment.vector.VectorColumnSelectorFactory), which offer precise control over how much memory is being used.

    • optimizeForSegment

      public AggregatorFactory optimizeForSegment(PerSegmentQueryOptimizationContext optimizationContext)
      Return a potentially optimized form of this AggregatorFactory for per-segment queries.

    • withName

      public AggregatorFactory withName(String newName)
      Used in cases where we want to change the output name of the aggregator to something else. For eg: if we have a query `select a, sum(b) as total group by a from table` the aggregator returned from the native group by query is "a0" set in org.apache.druid.sql.calcite.rel.DruidQuery#computeAggregations. We can use withName("total") to set the output name of the aggregator to "total".

      As all implementations of this interface method may not exist, callers of this method are advised to handle such a case.

      Parameters:
      newName - newName of the output for aggregator factory
      Returns:
      AggregatorFactory with the output name set as the input param.

    • substituteCombiningFactory

      @Nullable public AggregatorFactory substituteCombiningFactory(AggregatorFactory preAggregated)
      Check to see if we can make a 'combining' factory of this aggregator that is suitable to process input from a selector of values produced by the other AggregatorFactory representing pre-aggregated data. Typically, this means that this and the other aggregator have the same inputs (requiredFields(), and the same options for how the data was constructed into the intermediary type. If suitable, this method returns a 'combining' aggregator factory of this aggregator to use to process the pre-aggregated data which was produced by the other aggregator.

      This method is used indirectly in service of checking if a QueryableProjection can be used instead of the base table during CursorFactory.makeCursorHolder(CursorBuildSpec), which checks if this aggregator can be substituted for its combining aggregator if and only if there exists a column that a cursor can read which was created by an aggregator that satisfies this method. In other words, this aggregator is the 'query' aggregator defined on the CursorBuildSpec, the argument to this method is the aggregator which created some column whose selectors are available to the cursor. If all aggregators on the CursorBuildSpec can be paired with aggregators from the underlying table in the cursor factory, then CursorHolder.isPreAggregated() will be set to true indicating that query engines should use this substituted aggregator instead of the original aggregators.

      Parameters:
      preAggregated - AggregatorFactory which produced the partially aggregated values which are available in a selector
      Returns:
      a "combining" AggregatorFactory to use with the pre-aggregated selector data

    • mergeAggregators

      @Nullable public static AggregatorFactory[] mergeAggregators(List<AggregatorFactory[]> aggregatorsList)
      Merges the list of AggregatorFactory[] (presumable from metadata of some segments being merged) and returns merged AggregatorFactory[] (for the metadata for merged segment). Null is returned if it is not possible to do the merging for any of the following reason. - one of the element in input list is null i.e. aggregators for one the segments being merged is unknown - AggregatorFactory of same name can not be merged if they are not compatible
      Parameters:
      aggregatorsList -
      Returns:
      merged AggregatorFactory[] or Null if merging is not possible.