Package org.apache.druid.segment

Class AutoTypeColumnMerger

java.lang.Object

org.apache.druid.segment.AutoTypeColumnMerger

All Implemented Interfaces:

DimensionMerger, DimensionMergerV9

public class AutoTypeColumnMerger
extends Object
implements DimensionMergerV9

Column merger for AutoTypeColumnIndexer to eventually produce some form of NestedCommonFormatColumn.

Depending on the types of values encountered

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	AutoTypeColumnMerger.ArrayDictionaryMergingIterator
static class	AutoTypeColumnMerger.IdLookupArrayIterator

Field Summary

Fields

Modifier and Type	Field	Description
static final Comparator<com.google.common.collect.PeekingIterator<Double>>	DOUBLE_MERGING_COMPARATOR
static final Comparator<com.google.common.collect.PeekingIterator<Long>>	LONG_MERGING_COMPARATOR
static final Comparator<com.google.common.collect.PeekingIterator<String>>	STRING_MERGING_COMPARATOR

Constructor Summary

Constructors

Constructor	Description
AutoTypeColumnMerger(String name, String outputName, ColumnType castToType, NestedCommonFormatColumnFormatSpec columnFormatSpec, SegmentWriteOutMedium segmentWriteOutMedium, File segmentBaseDir, Closer closer)

Method Summary

Modifier and Type	Method	Description
void	attachParent(DimensionMergerV9 parent, List<IndexableAdapter> projectionAdapters)	Attaches the DimensionMergerV9 of a "projection" parent column so that stuff like value dictionaries can be shared between parent and child.
ColumnValueSelector	convertSortedSegmentRowValuesToMergedRowValues(int segmentIndex, ColumnValueSelector source)	Creates a value selector, which converts values with per-segment, _sorted order_ (see DimensionIndexer.convertUnsortedValuesToSorted(org.apache.druid.segment.ColumnValueSelector)) encoding from the given selector to their equivalent representation in the merged set of rows.
protected DictionaryIdLookup	getIdLookup()
boolean	hasOnlyNulls()	Returns true if this dimension has no data besides nulls.
ColumnDescriptor	makeColumnDescriptor()	Return a ColumnDescriptor containing ColumnPartSerde objects appropriate for this DimensionMerger's value metadata, sequence of row values, and index structures.
void	processMergedRow(ColumnValueSelector selector)	Process a column value(s) (potentially multi-value) of a row from the given selector and update the DimensionMerger's internal state.
void	writeIndexes(List<IntBuffer> segmentRowNumConversions)	Internally construct any index structures relevant to this DimensionMerger.
void	writeMergedValueDictionary(List<IndexableAdapter> adapters)	Given a list of segment adapters: - Read _sorted order_ (e.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.druid.segment.DimensionMergerV9

markAsParent

Field Details

STRING_MERGING_COMPARATOR

public static final Comparator<com.google.common.collect.PeekingIterator<String>> STRING_MERGING_COMPARATOR

LONG_MERGING_COMPARATOR

public static final Comparator<com.google.common.collect.PeekingIterator<Long>> LONG_MERGING_COMPARATOR

DOUBLE_MERGING_COMPARATOR

public static final Comparator<com.google.common.collect.PeekingIterator<Double>> DOUBLE_MERGING_COMPARATOR

Constructor Details

AutoTypeColumnMerger

public AutoTypeColumnMerger(String name,
 String outputName,
 @Nullable
 ColumnType castToType,
 NestedCommonFormatColumnFormatSpec columnFormatSpec,
 SegmentWriteOutMedium segmentWriteOutMedium,
 File segmentBaseDir,
 Closer closer)

Parameters:

name - column name

outputName - output smoosh file name. if this is a base table column, it will be the equivalent to name, however if this merger is for a projection, this will be prefixed with the projection name so that multiple projections can store the same column name at different smoosh file "paths"

castToType - optional mechanism to enforce that all values are a specific type

columnFormatSpec - column level storage options such as compression format and bitmap type

segmentWriteOutMedium - temporary storage location to stage segment outputs before finalizing into the segment

closer - resource closer if this merger needs to attach any closables that should be cleaned up when the segment is finished writing

Method Details

writeMergedValueDictionary

public void writeMergedValueDictionary(List<IndexableAdapter> adapters)
                                throws IOException

Description copied from interface: DimensionMerger

Given a list of segment adapters: - Read _sorted order_ (e. g. see IncrementalIndexAdapter.getDimValueLookup(String)) dictionary encoding information from the adapters - Merge those sorted order dictionary into a one big sorted order dictionary and write this merged dictionary. The implementer should maintain knowledge of the "index number" of the adapters in the input list, i.e., the position of each adapter in the input list. This "index number" will be used to refer to specific segments later in DimensionMerger.convertSortedSegmentRowValuesToMergedRowValues(int, org.apache.druid.segment.ColumnValueSelector).

Specified by:

writeMergedValueDictionary in interface DimensionMerger

Parameters:

adapters - List of adapters to be merged.

Throws:

IOException

See Also:

• DimensionIndexer.convertUnsortedValuesToSorted(org.apache.druid.segment.ColumnValueSelector)

convertSortedSegmentRowValuesToMergedRowValues

public ColumnValueSelector convertSortedSegmentRowValuesToMergedRowValues(int segmentIndex,
 ColumnValueSelector source)

Description copied from interface: DimensionMerger

Creates a value selector, which converts values with per-segment, _sorted order_ (see DimensionIndexer.convertUnsortedValuesToSorted(org.apache.druid.segment.ColumnValueSelector)) encoding from the given selector to their equivalent representation in the merged set of rows. This method is used by the index merging process to build the merged sequence of rows. The implementing class is expected to use the merged value metadata constructed during DimensionMerger.writeMergedValueDictionary(List), if applicable. For example, an implementation of this function for a dictionary-encoded String column would convert the segment-specific, sorted order dictionary values within the row to the common merged dictionary values determined during DimensionMerger.writeMergedValueDictionary(List).

Specified by:

convertSortedSegmentRowValuesToMergedRowValues in interface DimensionMerger

Parameters:

segmentIndex - indicates which segment the row originated from, in the order established in DimensionMerger.writeMergedValueDictionary(List)

source - the selector from which to take values to convert

Returns:

a selector with converted values

processMergedRow

public void processMergedRow(ColumnValueSelector selector)
                      throws IOException

Description copied from interface: DimensionMerger

Process a column value(s) (potentially multi-value) of a row from the given selector and update the DimensionMerger's internal state. After constructing a merged sequence of rows across segments, the index merging process will iterate through these rows and on each iteration, for each column, pass the column value selector to the corresponding DimensionMerger. This allows each DimensionMerger to build its internal view of the sequence of merged rows, to be written out to a segment later.

Specified by:

processMergedRow in interface DimensionMerger

Throws:

IOException

writeIndexes

public void writeIndexes(@Nullable
 List<IntBuffer> segmentRowNumConversions)

Description copied from interface: DimensionMerger

Internally construct any index structures relevant to this DimensionMerger. After receiving the sequence of merged rows via iterated DimensionMerger.processMergedRow(org.apache.druid.segment.ColumnValueSelector) calls, the DimensionMerger can now build any index structures it needs. For example, a dictionary encoded String implementation would create its bitmap indexes for the merged segment during this step. The index merger will provide a list of row number conversion IntBuffer objects. Each IntBuffer is associated with one of the segments being merged; the position of the IntBuffer in the list corresponds to the position of segment adapters within the input list of DimensionMerger.writeMergedValueDictionary(List). For example, suppose there are two segments A and B. Row 24 from segment A maps to row 99 in the merged sequence of rows, The IntBuffer for segment A would have a mapping of 24 -> 99.

Specified by:

writeIndexes in interface DimensionMerger

Parameters:

segmentRowNumConversions - A list of row number conversion IntBuffer objects.

hasOnlyNulls

public boolean hasOnlyNulls()

Description copied from interface: DimensionMerger

Returns true if this dimension has no data besides nulls. See NullColumnPartSerde for how null-only columns are stored in the segment.

Specified by:

hasOnlyNulls in interface DimensionMerger

makeColumnDescriptor

public ColumnDescriptor makeColumnDescriptor()

Description copied from interface: DimensionMergerV9

Return a ColumnDescriptor containing ColumnPartSerde objects appropriate for this DimensionMerger's value metadata, sequence of row values, and index structures.

Specified by:

makeColumnDescriptor in interface DimensionMergerV9

Returns:

ColumnDescriptor that IndexMergerV9 will use to build a column.

getIdLookup

protected DictionaryIdLookup getIdLookup()

attachParent

public void attachParent(DimensionMergerV9 parent,
 List<IndexableAdapter> projectionAdapters)
                  throws IOException

Description copied from interface: DimensionMergerV9

Attaches the DimensionMergerV9 of a "projection" parent column so that stuff like value dictionaries can be shared between parent and child. This method is called during merging instead of DimensionMerger.writeMergedValueDictionary(List) if the parent column exists.

Specified by:

attachParent in interface DimensionMergerV9

Throws:

IOException

See Also:

• IndexMergerV9.makeProjections(org.apache.druid.java.util.common.io.smoosh.FileSmoosher, java.util.List<org.apache.druid.segment.AggregateProjectionMetadata>, java.util.List<org.apache.druid.segment.IndexableAdapter>, org.apache.druid.segment.IndexSpec, org.apache.druid.segment.writeout.SegmentWriteOutMedium, org.apache.druid.segment.ProgressIndicator, java.io.File, org.apache.druid.java.util.common.io.Closer, java.util.Map<java.lang.String, org.apache.druid.segment.DimensionMergerV9>, org.apache.druid.segment.Metadata)