Package org.apache.druid.query.groupby.epinephelinae

Class ConcurrentGrouper<KeyType>

java.lang.Object

org.apache.druid.query.groupby.epinephelinae.ConcurrentGrouper<KeyType>

All Implemented Interfaces:

Closeable, AutoCloseable, Grouper<KeyType>

public class ConcurrentGrouper<KeyType>
extends Object
implements Grouper<KeyType>

Grouper based around a set of underlying SpillingGrouper instances. Thread-safe.

The passed-in buffer is cut up into concurrencyHint slices, and each slice is passed to a different underlying grouper. Access to each slice is separately synchronized. As long as the result set fits in memory, keys are partitioned between buffers based on their hash, and multiple threads can write into the same buffer. When it becomes clear that the result set does not fit in memory, the table switches to a mode where each thread gets its own buffer and its own spill files on disk.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.druid.query.groupby.epinephelinae.Grouper

Grouper.BufferComparator, Grouper.Entry<T>, Grouper.KeySerde<T>, Grouper.KeySerdeFactory<T>

Constructor Summary

Constructors

Constructor

Description

ConcurrentGrouper(GroupByQueryConfig groupByQueryConfig, com.google.common.base.Supplier<ByteBuffer> bufferSupplier, ReferenceCountingResourceHolder<ByteBuffer> combineBufferHolder, Grouper.KeySerdeFactory<KeyType> keySerdeFactory, Grouper.KeySerdeFactory<KeyType> combineKeySerdeFactory, ColumnSelectorFactory columnSelectorFactory, AggregatorFactory[] aggregatorFactories, LimitedTemporaryStorage temporaryStorage, com.fasterxml.jackson.databind.ObjectMapper spillMapper, int concurrencyHint, DefaultLimitSpec limitSpec, boolean sortHasNonGroupingFields, com.google.common.util.concurrent.ListeningExecutorService executor, int priority, boolean hasQueryTimeout, long queryTimeoutAt, GroupByStatsProvider.PerQueryStats perQueryStats)

Method Summary

Modifier and Type	Method	Description
AggregateResult	aggregate(KeyType key, int keyHash)	Aggregate the current row with the provided key.
void	close()	Close the grouper and release associated resources.
void	init()	Initialize the grouper.
boolean	isInitialized()	Check this grouper is initialized or not.
CloseableIterator<Grouper.Entry<KeyType>>	iterator(boolean sorted)	Iterate through entries.
void	reset()	Reset the grouper to its initial state.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.druid.query.groupby.epinephelinae.Grouper

aggregate, hashFunction

Constructor Details

ConcurrentGrouper

public ConcurrentGrouper(GroupByQueryConfig groupByQueryConfig,
 com.google.common.base.Supplier<ByteBuffer> bufferSupplier,
 @Nullable
 ReferenceCountingResourceHolder<ByteBuffer> combineBufferHolder,
 Grouper.KeySerdeFactory<KeyType> keySerdeFactory,
 Grouper.KeySerdeFactory<KeyType> combineKeySerdeFactory,
 ColumnSelectorFactory columnSelectorFactory,
 AggregatorFactory[] aggregatorFactories,
 LimitedTemporaryStorage temporaryStorage,
 com.fasterxml.jackson.databind.ObjectMapper spillMapper,
 int concurrencyHint,
 DefaultLimitSpec limitSpec,
 boolean sortHasNonGroupingFields,
 com.google.common.util.concurrent.ListeningExecutorService executor,
 int priority,
 boolean hasQueryTimeout,
 long queryTimeoutAt,
 GroupByStatsProvider.PerQueryStats perQueryStats)

Method Details

init

public void init()

Description copied from interface: Grouper

Initialize the grouper. This method needs to be called before calling Grouper.aggregate(Object) and Grouper.aggregate(Object, int).

Specified by:

init in interface Grouper<KeyType>

isInitialized

public boolean isInitialized()

Description copied from interface: Grouper

Check this grouper is initialized or not.

Specified by:

isInitialized in interface Grouper<KeyType>

Returns:

true if the grouper is already initialized, otherwise false.

aggregate

public AggregateResult aggregate(KeyType key,
 int keyHash)

Description copied from interface: Grouper

Aggregate the current row with the provided key. Some implementations are thread-safe and some are not.

Specified by:

aggregate in interface Grouper<KeyType>

Parameters:

key - key object

keyHash - result of Grouper.hashFunction() on the key

Returns:

result that is ok if the row was aggregated, not ok if a resource limit was hit

reset

public void reset()

Description copied from interface: Grouper

Reset the grouper to its initial state.

Specified by:

reset in interface Grouper<KeyType>

iterator

public CloseableIterator<Grouper.Entry<KeyType>> iterator(boolean sorted)

Description copied from interface: Grouper

Iterate through entries.

Some implementations allow writes even after this method is called. After you are done with the iterator returned by this method, you should either call Grouper.close() (if you are done with the Grouper) or Grouper.reset() (if you want to reuse it). Some implementations allow calling Grouper.iterator(boolean) again if you want another iterator. But, this method must not be called by multiple threads concurrently.

If "sorted" is true then the iterator will return sorted results. It will use KeyType's natural ordering on deserialized objects, and will use the Grouper.KeySerde.bufferComparator() on serialized objects. Woe be unto you if these comparators are not equivalent.

Callers must process and discard the returned Grouper.Entrys immediately because some implementations can reuse the key objects.

Specified by:

iterator in interface Grouper<KeyType>

Parameters:

sorted - return sorted results

Returns:

entry iterator

close

public void close()

Description copied from interface: Grouper

Close the grouper and release associated resources.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Specified by:

close in interface Grouper<KeyType>