Package org.apache.druid.data.input

Interface InputSource

All Known Subinterfaces:
SplittableInputSource<T>, SystemFieldInputSource
All Known Implementing Classes:
AbstractInputSource, CloudObjectInputSource, CombiningInputSource, HttpInputSource, InlineInputSource, LocalInputSource
public interface InputSource
InputSource abstracts the storage system where input data is stored. It creates an InputSourceReader to read data from the given input source. The most common use case would be: 

   InputSourceReader reader = inputSource.reader();
   try (CloseableIterator<InputRow> iterator = reader.read()) {
     while (iterator.hasNext()) {
       InputRow row = iterator.next();
       processRow(row);
     }
   }

  • Field Details

  • Method Details

    • isSplittable

      boolean isSplittable()
      Returns true if this inputSource can be processed in parallel using ParallelIndexSupervisorTask. It must be castable to SplittableInputSource and the various SplittableInputSource methods must work as documented.

    • needsFormat

      boolean needsFormat()
      Returns true if this inputSource supports different InputFormats. Some inputSources such as LocalInputSource can store files of any format. These storage types require an InputFormat to be passed so that InputSourceReader can parse data properly. However, some storage types have a fixed format. For example, druid inputSource always reads segments. These inputSources should return false for this method.

    • reader

      InputSourceReader reader(InputRowSchema inputRowSchema, @Nullable InputFormat inputFormat, File temporaryDirectory)
      Creates an InputSourceReader.
      Parameters:
      inputRowSchema - for InputRow
      inputFormat - to parse data. It can be null if needsFormat() = true
      temporaryDirectory - to store temp data. It will be cleaned up automatically once the task is finished.

    • getTypes

      @Nonnull default Set<String> getTypes()
      The types of input sources uses. A set is returned here, as some InputSource implementation allow for combining of multiple input sources.
      Returns:
      The types of input sources uses