Package io.delta.kernel.defaults.internal.parquet

Class ParquetFileReader.RowRecordCollector

Object
org.apache.parquet.io.api.RecordMaterializer<Object>
io.delta.kernel.defaults.internal.parquet.ParquetFileReader.RowRecordCollector
Enclosing class:
ParquetFileReader
public static class ParquetFileReader.RowRecordCollector extends org.apache.parquet.io.api.RecordMaterializer<Object>
Collects the records given by the Parquet reader as columnar data. Parquet reader allows reading data row by row, but ParquetFileReader wants to expose the data as a columnar batch. Parquet reader takes an implementation of RecordMaterializer to which it gives data for each column one row at a time. This RecordMaterializer implementation collects the column values for multiple rows and returns a ColumnarBatch at the end.

  • Nested Class Summary

    Nested classes/interfaces inherited from class org.apache.parquet.io.api.RecordMaterializer

    org.apache.parquet.io.api.RecordMaterializer.RecordMaterializationException

  • Constructor Summary

    Constructors
    Constructor
    Description
    RowRecordCollector(int maxBatchSize, io.delta.kernel.types.StructType readSchema, org.apache.parquet.schema.MessageType fileSchema)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    finalizeCurrentRow(long fileRowIndex)
    Finalize the current row.
    Object
    getCurrentRecord()
    Return a fake object.
    io.delta.kernel.data.ColumnarBatch
    getDataAsColumnarBatch(int batchSize)
    Return the data collected so far as a ColumnarBatch.
    org.apache.parquet.io.api.GroupConverter
    getRootConverter()
     
    void
    skipCurrentRecord()
     

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • RowRecordCollector

      public RowRecordCollector(int maxBatchSize, io.delta.kernel.types.StructType readSchema, org.apache.parquet.schema.MessageType fileSchema)

  • Method Details

    • skipCurrentRecord

      public void skipCurrentRecord()
      Overrides:
      skipCurrentRecord in class org.apache.parquet.io.api.RecordMaterializer<Object>

    • getCurrentRecord

      public Object getCurrentRecord()
      Return a fake object. This is not used by ParquetFileReader, instead getDataAsColumnarBatch(int)} once a sufficient number of rows are collected.
      Specified by:
      getCurrentRecord in class org.apache.parquet.io.api.RecordMaterializer<Object>

    • getRootConverter

      public org.apache.parquet.io.api.GroupConverter getRootConverter()
      Specified by:
      getRootConverter in class org.apache.parquet.io.api.RecordMaterializer<Object>

    • getDataAsColumnarBatch

      public io.delta.kernel.data.ColumnarBatch getDataAsColumnarBatch(int batchSize)
      Return the data collected so far as a ColumnarBatch.

    • finalizeCurrentRow

      public void finalizeCurrentRow(long fileRowIndex)
      Finalize the current row.
      Parameters:
      fileRowIndex - the file row index of the row just processed