Package org.apache.druid.segment.incremental

Class IncrementalIndex

java.lang.Object

org.apache.druid.segment.incremental.IncrementalIndex

All Implemented Interfaces:

Closeable, AutoCloseable, Iterable<Row>, Expr.InputBindingInspector, ColumnInspector, IncrementalIndexRowSelector

Direct Known Subclasses:

OnheapIncrementalIndex

public abstract class IncrementalIndex
extends Object
implements IncrementalIndexRowSelector, ColumnInspector, Iterable<Row>, Closeable

In-memory, row-based data structure used to hold data during ingestion. Realtime tasks query this index using IncrementalIndexCursorFactory. Concurrency model: add(InputRow) is not thread-safe, and must be called from a single thread or externally synchronized. However, the methods that support IncrementalIndexCursorFactory are thread-safe, and may be called concurrently with each other, and with the "add" methods. This concurrency model supports real-time queries of the data in the index.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	IncrementalIndex.AddToFactsResult
static final class	IncrementalIndex.DimensionDesc
static class	IncrementalIndex.InputRowHolder
static final class	IncrementalIndex.MetricDesc

Field Summary

Fields

Modifier and Type	Field	Description
protected final IncrementalIndex.InputRowHolder	inputRowHolder
protected final boolean	preserveExistingMetrics
protected final int	timePosition	Position of ColumnHolder.TIME_COLUMN_NAME in the sort order, relative to elements of dimensionDescs.

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	IncrementalIndex(IncrementalIndexSchema incrementalIndexSchema, boolean preserveExistingMetrics)

Method Summary

Modifier and Type	Method	Description
IncrementalIndexAddResult	add(InputRow row)	Adds a new row.
protected abstract IncrementalIndex.AddToFactsResult	addToFacts(IncrementalIndexRow key, IncrementalIndex.InputRowHolder inputRowHolder)
abstract boolean	canAppendRow()
void	close()
protected final Comparator<IncrementalIndexRow>	dimsComparator()
InputRow	formatRow(InputRow row)
AtomicLong	getBytesInMemory()
ColumnCapabilities	getColumnCapabilities(String columnName)	Returns capabilities of a particular column.
ColumnFormat	getColumnFormat(String columnName)	Get ColumnFormat for a dimension, metrics, or time column, or null if the column does not exist
Map<String,ColumnFormat>	getColumnFormats()
List<String>	getColumnNames()	Returns all column names, including ColumnHolder.TIME_COLUMN_NAME.
static ParseException	getCombinedParseException(InputRow row, List<String> dimParseExceptionMessages, List<String> aggParseExceptionMessages)
IncrementalIndex.DimensionDesc	getDimension(String dimension)	Returns the descriptor for a particular dimension.
List<String>	getDimensionNames(boolean includeTime)	Returns names of dimension columns.
List<String>	getDimensionOrder()	Returns names of time and dimension columns, in persist sort order.
List<IncrementalIndex.DimensionDesc>	getDimensions()	Returns a descriptor for each dimension.
DimensionsSpec	getDimensionsSpec()	Returns dimensionsSpec from the ingestionSpec.
org.joda.time.Interval	getInterval()
org.joda.time.DateTime	getMaxIngestedEventTime()
org.joda.time.DateTime	getMaxTime()
Metadata	getMetadata()
IncrementalIndex.MetricDesc	getMetric(String metric)	Get IncrementalIndex.MetricDesc which provides column capabilities and position in the aggregators section of the row
AggregatorFactory[]	getMetricAggs()
List<String>	getMetricNames()	Returns list of all metric column names for the row selector
org.joda.time.DateTime	getMinTime()
List<OrderBy>	getOrdering()	Ordering for the data in the facts table
abstract String	getOutOfRowsReason()
abstract IncrementalIndexRowSelector	getProjection(String name)
abstract QueryableProjection<IncrementalIndexRowSelector>	getProjection(CursorBuildSpec buildSpec)
int	getTimePosition()	Position of the time column in IncrementalIndexRowSelector.getOrdering()
protected abstract void	initAggs(AggregatorFactory[] metrics, IncrementalIndex.InputRowHolder rowSupplier)
boolean	isEmpty()	Are there any IncrementalIndexRow stored in the FactsHolder?
boolean	isRollup()
abstract Iterable<Row>	iterableWithPostAggregations(List<PostAggregator> postAggs, boolean descending)
Iterator<Row>	iterator()
void	loadDimensionIterable(Iterable<String> oldDimensionOrder, Map<String,ColumnFormat> oldColumnFormats)	Currently called to initialize IncrementalIndex dimension order during index creation Index dimension ordering could be changed to initialize from DimensionsSpec after resolution of https://github.com/apache/druid/issues/2011
protected ColumnSelectorFactory	makeColumnSelectorFactory(AggregatorFactory agg, IncrementalIndex.InputRowHolder in)
static ColumnSelectorFactory	makeColumnSelectorFactory(VirtualColumns virtualColumns, IncrementalIndex.InputRowHolder inputRowHolder, AggregatorFactory agg)	Column selector used at ingestion time for inputs to aggregators.
static ColumnCapabilitiesImpl	makeDefaultCapabilitiesFromValueType(ColumnType type)
static ColumnValueSelector<?>	makeMetricColumnValueSelector(IncrementalIndexRowSelector rowSelector, IncrementalIndexRowHolder currEntry, String metric)
int	numRows()	Number of rows in FactsHolder

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.druid.segment.ColumnInspector

getType

Methods inherited from interface org.apache.druid.math.expr.Expr.InputBindingInspector

areNumeric, areNumeric, areSameTypes, areSameTypes, areScalar, areScalar, canVectorize, canVectorize

Methods inherited from interface org.apache.druid.segment.incremental.IncrementalIndexRowSelector

getFacts, getLastRowIndex, getMetricDoubleValue, getMetricFloatValue, getMetricLongValue, getMetricObjectValue, isNull

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Field Details

preserveExistingMetrics

protected final boolean preserveExistingMetrics

timePosition

protected final int timePosition

Position of ColumnHolder.TIME_COLUMN_NAME in the sort order, relative to elements of dimensionDescs. For example, for the sort order [x, __time, y], dimensionDescs contains [x, y] and timePosition is 1.

inputRowHolder

protected final IncrementalIndex.InputRowHolder inputRowHolder

Constructor Details

IncrementalIndex

protected IncrementalIndex(IncrementalIndexSchema incrementalIndexSchema,
 boolean preserveExistingMetrics)

Parameters:

incrementalIndexSchema - the schema to use for incremental index

preserveExistingMetrics - When set to true, for any row that already has metric (with the same name defined in metricSpec), the metric aggregator in metricSpec is skipped and the existing metric is unchanged. If the row does not already have the metric, then the metric aggregator is applied on the source column as usual. This should only be set for DruidInputSource since that is the only case where we can have existing metrics. This is currently only use by auto compaction and should not be use for anything else.

Method Details

makeColumnSelectorFactory

public static ColumnSelectorFactory makeColumnSelectorFactory(VirtualColumns virtualColumns,
 IncrementalIndex.InputRowHolder inputRowHolder,
 @Nullable
 AggregatorFactory agg)

Column selector used at ingestion time for inputs to aggregators.

Parameters:

virtualColumns - virtual columns

inputRowHolder - ingestion-time input row holder

agg - the aggregator, or null to make a generic aggregator. Only required if the agg has AggregatorFactory.getIntermediateType() as ValueType.COMPLEX, because in this case we need to do some magic to ensure the correct values show up.

Returns:

column selector factory

getProjection

@Nullable
public abstract QueryableProjection<IncrementalIndexRowSelector> getProjection(CursorBuildSpec buildSpec)

getProjection

public abstract IncrementalIndexRowSelector getProjection(String name)

canAppendRow

public abstract boolean canAppendRow()

getOutOfRowsReason

public abstract String getOutOfRowsReason()

initAggs

protected abstract void initAggs(AggregatorFactory[] metrics,
 IncrementalIndex.InputRowHolder rowSupplier)

addToFacts

protected abstract IncrementalIndex.AddToFactsResult addToFacts(IncrementalIndexRow key,
 IncrementalIndex.InputRowHolder inputRowHolder)

iterableWithPostAggregations

public abstract Iterable<Row> iterableWithPostAggregations(@Nullable
 List<PostAggregator> postAggs,
 boolean descending)

isRollup

public boolean isRollup()

close

public void close()

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

formatRow

public InputRow formatRow(InputRow row)

getColumnFormats

public Map<String,ColumnFormat> getColumnFormats()

getColumnCapabilities

@Nullable
public ColumnCapabilities getColumnCapabilities(String columnName)

Description copied from interface: ColumnInspector

Returns capabilities of a particular column.

Specified by:

getColumnCapabilities in interface ColumnInspector

Parameters:

columnName - column name

Returns:

capabilities, or null

getColumnFormat

@Nullable
public ColumnFormat getColumnFormat(String columnName)

Description copied from interface: IncrementalIndexRowSelector

Get ColumnFormat for a dimension, metrics, or time column, or null if the column does not exist

Specified by:

getColumnFormat in interface IncrementalIndexRowSelector

add

public IncrementalIndexAddResult add(InputRow row)

Adds a new row. The row might correspond with another row that already exists, in which case this will update that row instead of inserting a new one. Not thread-safe.

Parameters:

row - the row of data to add

Returns:

the number of rows in the data set after adding the InputRow. If any parse failure occurs, a ParseException is returned in IncrementalIndexAddResult.

getCombinedParseException

@Nullable
public static ParseException getCombinedParseException(InputRow row,
 @Nullable
 List<String> dimParseExceptionMessages,
 @Nullable
 List<String> aggParseExceptionMessages)

isEmpty

public boolean isEmpty()

Description copied from interface: IncrementalIndexRowSelector

Are there any IncrementalIndexRow stored in the FactsHolder?

Specified by:

isEmpty in interface IncrementalIndexRowSelector

numRows

public int numRows()

Description copied from interface: IncrementalIndexRowSelector

Number of rows in FactsHolder

Specified by:

numRows in interface IncrementalIndexRowSelector

getBytesInMemory

public AtomicLong getBytesInMemory()

getMetricAggs

public AggregatorFactory[] getMetricAggs()

getDimensionsSpec

public DimensionsSpec getDimensionsSpec()

Returns dimensionsSpec from the ingestionSpec.

getDimensionNames

public List<String> getDimensionNames(boolean includeTime)

Returns names of dimension columns.

Specified by:

getDimensionNames in interface IncrementalIndexRowSelector

Parameters:

includeTime - whether to include ColumnHolder.TIME_COLUMN_NAME.

getDimensions

public List<IncrementalIndex.DimensionDesc> getDimensions()

Returns a descriptor for each dimension. Does not inclue ColumnHolder.TIME_COLUMN_NAME.

Specified by:

getDimensions in interface IncrementalIndexRowSelector

getDimension

@Nullable
public IncrementalIndex.DimensionDesc getDimension(String dimension)

Returns the descriptor for a particular dimension.

Specified by:

getDimension in interface IncrementalIndexRowSelector

getMetric

@Nullable
public IncrementalIndex.MetricDesc getMetric(String metric)

Description copied from interface: IncrementalIndexRowSelector

Get IncrementalIndex.MetricDesc which provides column capabilities and position in the aggregators section of the row

Specified by:

getMetric in interface IncrementalIndexRowSelector

getOrdering

public List<OrderBy> getOrdering()

Description copied from interface: IncrementalIndexRowSelector

Ordering for the data in the facts table

Specified by:

getOrdering in interface IncrementalIndexRowSelector

getTimePosition

public int getTimePosition()

Description copied from interface: IncrementalIndexRowSelector

Position of the time column in IncrementalIndexRowSelector.getOrdering()

Specified by:

getTimePosition in interface IncrementalIndexRowSelector

makeMetricColumnValueSelector

public static ColumnValueSelector<?> makeMetricColumnValueSelector(IncrementalIndexRowSelector rowSelector,
 IncrementalIndexRowHolder currEntry,
 String metric)

getInterval

public org.joda.time.Interval getInterval()

getMinTime

@Nullable
public org.joda.time.DateTime getMinTime()

getMaxTime

@Nullable
public org.joda.time.DateTime getMaxTime()

getDimensionOrder

public List<String> getDimensionOrder()

Returns names of time and dimension columns, in persist sort order. Includes ColumnHolder.TIME_COLUMN_NAME.

makeDefaultCapabilitiesFromValueType

public static ColumnCapabilitiesImpl makeDefaultCapabilitiesFromValueType(ColumnType type)

loadDimensionIterable

public void loadDimensionIterable(Iterable<String> oldDimensionOrder,
 Map<String,ColumnFormat> oldColumnFormats)

Currently called to initialize IncrementalIndex dimension order during index creation Index dimension ordering could be changed to initialize from DimensionsSpec after resolution of https://github.com/apache/druid/issues/2011

Parameters:

oldDimensionOrder - dimension order to initialize

oldColumnFormats - formats for the dimensions

getMetricNames

public List<String> getMetricNames()

Description copied from interface: IncrementalIndexRowSelector

Returns list of all metric column names for the row selector

Specified by:

getMetricNames in interface IncrementalIndexRowSelector

getColumnNames

public List<String> getColumnNames()

Returns all column names, including ColumnHolder.TIME_COLUMN_NAME.

getMetadata

public Metadata getMetadata()

iterator

public Iterator<Row> iterator()

Specified by:

iterator in interface Iterable<Row>

getMaxIngestedEventTime

public org.joda.time.DateTime getMaxIngestedEventTime()

makeColumnSelectorFactory

protected ColumnSelectorFactory makeColumnSelectorFactory(@Nullable
 AggregatorFactory agg,
 IncrementalIndex.InputRowHolder in)

dimsComparator

protected final Comparator<IncrementalIndexRow> dimsComparator()