Package org.apache.druid.data.input.impl

Class CombiningInputSource

java.lang.Object

org.apache.druid.data.input.AbstractInputSource

org.apache.druid.data.input.impl.CombiningInputSource

All Implemented Interfaces:

SplittableInputSource, InputSource

public class CombiningInputSource
extends AbstractInputSource
implements SplittableInputSource

InputSource that combines data from multiple inputSources. The delegate inputSources must be splittable. The splits for this inputSource are created from the SplittableInputSource.createSplits(org.apache.druid.data.input.InputFormat, org.apache.druid.data.input.SplitHintSpec) of the delegate inputSources. Each inputSplit is paired up with its respective delegate inputSource so that during split, SplittableInputSource.withSplit(org.apache.druid.data.input.InputSplit<T>)is called against the correct inputSource for each inputSplit. This inputSource presently only supports a single InputFormat.

Field Summary

Fields

Modifier and Type	Field	Description
static final String	TYPE_KEY

Fields inherited from interface org.apache.druid.data.input.InputSource

TYPE_PROPERTY

Fields inherited from interface org.apache.druid.data.input.impl.SplittableInputSource

DEFAULT_SPLIT_HINT_SPEC

Constructor Summary

Constructors

Constructor	Description
CombiningInputSource(List<SplittableInputSource> delegates)

Method Summary

Modifier and Type	Method	Description
Stream<InputSplit>	createSplits(InputFormat inputFormat, SplitHintSpec splitHintSpec)	Creates a Stream of InputSplits.
boolean	equals(Object o)
int	estimateNumSplits(InputFormat inputFormat, SplitHintSpec splitHintSpec)	Returns an estimated total number of splits to be created via SplittableInputSource.createSplits(org.apache.druid.data.input.InputFormat, org.apache.druid.data.input.SplitHintSpec).
List<SplittableInputSource>	getDelegates()
Set<String>	getTypes()	The types of input sources uses.
int	hashCode()
boolean	needsFormat()	Returns true if this inputSource supports different InputFormats.
InputSource	withSplit(InputSplit split)	Helper method for ParallelIndexSupervisorTask.

Methods inherited from class org.apache.druid.data.input.AbstractInputSource

fixedFormatReader, formattableReader, reader

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.druid.data.input.InputSource

reader

Methods inherited from interface org.apache.druid.data.input.impl.SplittableInputSource

getSplitHintSpecOrDefault, isSplittable

Field Details

TYPE_KEY

public static final String TYPE_KEY

See Also:

• Constant Field Values

Constructor Details

CombiningInputSource

public CombiningInputSource(List<SplittableInputSource> delegates)

Method Details

getTypes

@Nonnull
public Set<String> getTypes()

Description copied from interface: InputSource

The types of input sources uses. A set is returned here, as some InputSource implementation allow for combining of multiple input sources.

Specified by:

getTypes in interface InputSource

Returns:

The types of input sources uses

getDelegates

public List<SplittableInputSource> getDelegates()

createSplits

public Stream<InputSplit> createSplits(InputFormat inputFormat,
 @Nullable
 SplitHintSpec splitHintSpec)

Description copied from interface: SplittableInputSource

Creates a Stream of InputSplits. The returned stream is supposed to be evaluated lazily to avoid consuming too much memory. Note that this interface also has SplittableInputSource.estimateNumSplits(org.apache.druid.data.input.InputFormat, org.apache.druid.data.input.SplitHintSpec) which is related to this method. The implementations should be careful to NOT cache the created splits in memory. Implementations can consider InputFormat.isSplittable() and SplitHintSpec to create splits in the same way with SplittableInputSource.estimateNumSplits(org.apache.druid.data.input.InputFormat, org.apache.druid.data.input.SplitHintSpec).

Specified by:

createSplits in interface SplittableInputSource

estimateNumSplits

public int estimateNumSplits(InputFormat inputFormat,
 @Nullable
 SplitHintSpec splitHintSpec)

Description copied from interface: SplittableInputSource

Returns an estimated total number of splits to be created via SplittableInputSource.createSplits(org.apache.druid.data.input.InputFormat, org.apache.druid.data.input.SplitHintSpec). The estimated number of splits doesn't have to be accurate and can be different from the actual number of InputSplits returned from SplittableInputSource.createSplits(org.apache.druid.data.input.InputFormat, org.apache.druid.data.input.SplitHintSpec). This will be used to estimate the progress of a phase in parallel indexing. See TaskMonitor for more details of the progress estimation. This method can be expensive if an implementation iterates all directories or whatever substructure to find all input entities. Implementations can consider InputFormat.isSplittable() and SplitHintSpec to find splits in the same way with SplittableInputSource.createSplits(org.apache.druid.data.input.InputFormat, org.apache.druid.data.input.SplitHintSpec).

Specified by:

estimateNumSplits in interface SplittableInputSource

withSplit

public InputSource withSplit(InputSplit split)

Description copied from interface: SplittableInputSource

Helper method for ParallelIndexSupervisorTask. Most of implementations can simply create a new instance with the given split.

Specified by:

withSplit in interface SplittableInputSource

needsFormat

public boolean needsFormat()

Description copied from interface: InputSource

Returns true if this inputSource supports different InputFormats. Some inputSources such as LocalInputSource can store files of any format. These storage types require an InputFormat to be passed so that InputSourceReader can parse data properly. However, some storage types have a fixed format. For example, druid inputSource always reads segments. These inputSources should return false for this method.

Specified by:

needsFormat in interface InputSource

equals

public boolean equals(Object o)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object