Package org.apache.druid.query.groupby

Class GroupingEngine

java.lang.Object

org.apache.druid.query.groupby.GroupingEngine

public class GroupingEngine
extends Object

Common code for processing GroupByQuery.

Field Summary

Fields

Modifier and Type	Field	Description
static final String	CTX_KEY_FUDGE_TIMESTAMP
static final String	CTX_KEY_OUTERMOST

Constructor Summary

Constructors

Constructor	Description
GroupingEngine(DruidProcessingConfig processingConfig, com.google.common.base.Supplier<GroupByQueryConfig> configSupplier, GroupByResourcesReservationPool groupByResourcesReservationPool, com.fasterxml.jackson.databind.ObjectMapper jsonMapper, com.fasterxml.jackson.databind.ObjectMapper spillMapper, QueryWatcher queryWatcher, GroupByStatsProvider groupByStatsProvider)

Method Summary

Modifier and Type	Method	Description
Sequence<ResultRow>	applyPostProcessing(Sequence<ResultRow> results, GroupByQuery query)	Apply the GroupByQuery "postProcessingFn", which is responsible for HavingSpec and LimitSpec.
static void	convertRowTypesToOutputTypes(List<DimensionSpec> dimensionSpecs, ResultRow resultRow, int resultRowDimensionStart)
BinaryOperator<ResultRow>	createMergeFn(Query<ResultRow> queryParam)	See QueryToolChest.createMergeFn(Query) for details, allows GroupByQueryQueryToolChest to delegate implementation to the strategy
Comparator<ResultRow>	createResultComparator(Query<ResultRow> queryParam)	See QueryToolChest.createResultComparator(Query), allows GroupByQueryQueryToolChest to delegate implementation to the strategy
static int	getCardinalityForArrayAggregation(GroupByQueryConfig querySpecificConfig, GroupByQuery query, ColumnInspector columnInspector, List<? extends GroupingSelector> groupingSelectors, ByteBuffer buffer)	Returns the cardinality of array needed to do array-based aggregation, or -1 if array-based aggregation is impossible.
static CursorBuildSpec	makeCursorBuildSpec(GroupByQuery query, QueryMetrics<?> queryMetrics)
Sequence<ResultRow>	mergeResults(QueryRunner<ResultRow> baseRunner, GroupByQuery query, ResponseContext responseContext)	Runs a provided QueryRunner on a provided GroupByQuery, which is assumed to return rows that are properly sorted (by timestamp and dimensions) but not necessarily fully merged (that is, there may be adjacent rows with the same timestamp and dimensions) and without PostAggregators computed.
QueryRunner<ResultRow>	mergeRunners(QueryProcessingPool queryProcessingPool, Iterable<QueryRunner<ResultRow>> queryRunners)	Merges a variety of single-segment query runners into a combined runner.
GroupByQuery	prepareGroupByQuery(GroupByQuery query)
static GroupByQueryResources	prepareResource(GroupByQuery query, BlockingPool<ByteBuffer> mergeBufferPool, boolean usesGroupByMergingQueryRunner, GroupByQueryConfig groupByQueryConfig)	Initializes resources required to run GroupByQueryQueryToolChest.mergeResults(QueryRunner) and GroupByMergingQueryRunner for a particular query.
Sequence<ResultRow>	process(GroupByQuery query, CursorFactory cursorFactory, TimeBoundaryInspector timeBoundaryInspector, NonBlockingPool<ByteBuffer> bufferPool, GroupByQueryMetrics groupByQueryMetrics)	Process a groupBy query on a single CursorFactory.
Sequence<ResultRow>	processSubqueryResult(GroupByQuery subquery, GroupByQuery query, GroupByQueryResources resource, Sequence<ResultRow> subqueryResult, boolean wasQueryPushedDown, GroupByStatsProvider.PerQueryStats perQueryStats)	Called by GroupByQueryQueryToolChest.mergeResults(QueryRunner) when it needs to process a subquery.
Sequence<ResultRow>	processSubtotalsSpec(GroupByQuery query, GroupByQueryResources resource, Sequence<ResultRow> queryResult, GroupByStatsProvider.PerQueryStats perQueryStats)	Called by GroupByQueryQueryToolChest.mergeResults(QueryRunner) when it needs to generate subtotals.
static boolean	summaryRowPreconditions(GroupByQuery query)	Whether a query should include a summary row.
static Sequence<ResultRow>	wrapSummaryRowIfNeeded(GroupByQuery query, Sequence<ResultRow> process)	Wraps the sequence around if for this query a summary row might be needed in case the input becomes empty.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

CTX_KEY_FUDGE_TIMESTAMP

public static final String CTX_KEY_FUDGE_TIMESTAMP

See Also:

• Constant Field Values

CTX_KEY_OUTERMOST

public static final String CTX_KEY_OUTERMOST

See Also:

• Constant Field Values

Constructor Details

GroupingEngine

@Inject
public GroupingEngine(DruidProcessingConfig processingConfig,
 com.google.common.base.Supplier<GroupByQueryConfig> configSupplier,
 GroupByResourcesReservationPool groupByResourcesReservationPool,
 com.fasterxml.jackson.databind.ObjectMapper jsonMapper,
 com.fasterxml.jackson.databind.ObjectMapper spillMapper,
 QueryWatcher queryWatcher,
 GroupByStatsProvider groupByStatsProvider)

Method Details

prepareResource

public static GroupByQueryResources prepareResource(GroupByQuery query,
 BlockingPool<ByteBuffer> mergeBufferPool,
 boolean usesGroupByMergingQueryRunner,
 GroupByQueryConfig groupByQueryConfig)

Initializes resources required to run GroupByQueryQueryToolChest.mergeResults(QueryRunner) and GroupByMergingQueryRunner for a particular query. The resources are to be acquired once throughout the execution of the query, or need to be re-acquired (if needed). Users must ensure that throughout the execution, a query already holding the resources shouldn't request for more resources, because that can cause deadlocks.

This method throws an exception if it is not able to allocate sufficient resources required for the query to succeed

createResultComparator

public Comparator<ResultRow> createResultComparator(Query<ResultRow> queryParam)

See QueryToolChest.createResultComparator(Query), allows GroupByQueryQueryToolChest to delegate implementation to the strategy

createMergeFn

public BinaryOperator<ResultRow> createMergeFn(Query<ResultRow> queryParam)

See QueryToolChest.createMergeFn(Query) for details, allows GroupByQueryQueryToolChest to delegate implementation to the strategy

prepareGroupByQuery

public GroupByQuery prepareGroupByQuery(GroupByQuery query)

mergeResults

public Sequence<ResultRow> mergeResults(QueryRunner<ResultRow> baseRunner,
 GroupByQuery query,
 ResponseContext responseContext)

Runs a provided QueryRunner on a provided GroupByQuery, which is assumed to return rows that are properly sorted (by timestamp and dimensions) but not necessarily fully merged (that is, there may be adjacent rows with the same timestamp and dimensions) and without PostAggregators computed. This method will fully merge the rows, apply PostAggregators, and return the resulting Sequence. The query will be modified using prepareGroupByQuery(GroupByQuery) before passing it down to the base runner. For example, "having" clauses will be removed and various context parameters will be adjusted. Despite the similar name, this method is much reduced in scope compared to GroupByQueryQueryToolChest.mergeResults(QueryRunner). That method does delegate to this one at some points, but has a truckload of other responsibility, including computing outer query results (if there are subqueries), computing subtotals (like GROUPING SETS), and computing the havingSpec and limitSpec.

Parameters:

baseRunner - base query runner

query - the groupBy query to run inside the base query runner

responseContext - the response context to pass to the base query runner

Returns:

merged result sequence

mergeRunners

public QueryRunner<ResultRow> mergeRunners(QueryProcessingPool queryProcessingPool,
 Iterable<QueryRunner<ResultRow>> queryRunners)

Merges a variety of single-segment query runners into a combined runner. Used by GroupByQueryRunnerFactory.mergeRunners(QueryProcessingPool, Iterable). In that sense, it is intended to go along with process(org.apache.druid.query.groupby.GroupByQuery, org.apache.druid.segment.CursorFactory, org.apache.druid.segment.TimeBoundaryInspector, org.apache.druid.collections.NonBlockingPool<java.nio.ByteBuffer>, org.apache.druid.query.groupby.GroupByQueryMetrics) (the runners created by that method will be fed into this method). This is primarily called on the data servers, to merge the results from processing on the segments. This method can also be called on the brokers if the query is operating on the local data sources, like the inline datasources. It uses GroupByMergingQueryRunner which requires the merge buffers to be passed in the responseContext of the query that is run.

Parameters:

queryProcessingPool - QueryProcessingPool service used for parallel execution of the query runners

queryRunners - collection of query runners to merge

Returns:

merged query runner

process

public Sequence<ResultRow> process(GroupByQuery query,
 CursorFactory cursorFactory,
 @Nullable
 TimeBoundaryInspector timeBoundaryInspector,
 NonBlockingPool<ByteBuffer> bufferPool,
 @Nullable
 GroupByQueryMetrics groupByQueryMetrics)

Process a groupBy query on a single CursorFactory. This is used by GroupByQueryRunnerFactory.createRunner(org.apache.druid.segment.Segment) to create per-segment QueryRunners. This method is only called on data servers, like Historicals (not the Broker).

Parameters:

query - the groupBy query

cursorFactory - cursor factory for the segment in question

timeBoundaryInspector - time boundary inspector for the segment in question

bufferPool - processing buffer pool

groupByQueryMetrics - metrics instance, will be populated if nonnull

Returns:

result sequence for the cursor factory

applyPostProcessing

public Sequence<ResultRow> applyPostProcessing(Sequence<ResultRow> results,
 GroupByQuery query)

Apply the GroupByQuery "postProcessingFn", which is responsible for HavingSpec and LimitSpec.

Parameters:

results - sequence of results

query - the groupBy query

Returns:

post-processed results, with HavingSpec and LimitSpec applied

processSubqueryResult

public Sequence<ResultRow> processSubqueryResult(GroupByQuery subquery,
 GroupByQuery query,
 GroupByQueryResources resource,
 Sequence<ResultRow> subqueryResult,
 boolean wasQueryPushedDown,
 GroupByStatsProvider.PerQueryStats perQueryStats)

Called by GroupByQueryQueryToolChest.mergeResults(QueryRunner) when it needs to process a subquery.

Parameters:

subquery - inner query

query - outer query

resource - resources returned by prepareResource(GroupByQuery, BlockingPool, boolean, GroupByQueryConfig)

subqueryResult - result rows from the subquery

wasQueryPushedDown - true if the outer query was pushed down (so we only need to merge the outer query's results, not run it from scratch like a normal outer query)

Returns:

results of the outer query

processSubtotalsSpec

public Sequence<ResultRow> processSubtotalsSpec(GroupByQuery query,
 GroupByQueryResources resource,
 Sequence<ResultRow> queryResult,
 GroupByStatsProvider.PerQueryStats perQueryStats)

Called by GroupByQueryQueryToolChest.mergeResults(QueryRunner) when it needs to generate subtotals.

Parameters:

query - query that has a "subtotalsSpec"

resource - resources returned by prepareResource(GroupByQuery, BlockingPool, boolean, GroupByQueryConfig)

queryResult - result rows from the main query

Returns:

results for each list of subtotals in the query, concatenated together

makeCursorBuildSpec

public static CursorBuildSpec makeCursorBuildSpec(GroupByQuery query,
 @Nullable
 QueryMetrics<?> queryMetrics)

getCardinalityForArrayAggregation

public static int getCardinalityForArrayAggregation(GroupByQueryConfig querySpecificConfig,
 GroupByQuery query,
 ColumnInspector columnInspector,
 List<? extends GroupingSelector> groupingSelectors,
 ByteBuffer buffer)

Returns the cardinality of array needed to do array-based aggregation, or -1 if array-based aggregation is impossible.

convertRowTypesToOutputTypes

public static void convertRowTypesToOutputTypes(List<DimensionSpec> dimensionSpecs,
 ResultRow resultRow,
 int resultRowDimensionStart)

wrapSummaryRowIfNeeded

public static Sequence<ResultRow> wrapSummaryRowIfNeeded(GroupByQuery query,
 Sequence<ResultRow> process)

Wraps the sequence around if for this query a summary row might be needed in case the input becomes empty.

summaryRowPreconditions

public static boolean summaryRowPreconditions(GroupByQuery query)

Whether a query should include a summary row. True for queries that correspond to SQL GROUP BY ().