Package io.delta.kernel.defaults.engine

Class DefaultParquetHandler

Object

io.delta.kernel.defaults.engine.DefaultParquetHandler

All Implemented Interfaces:

io.delta.kernel.engine.ParquetHandler

public class DefaultParquetHandler
extends Object
implements io.delta.kernel.engine.ParquetHandler

Default implementation of ParquetHandler based on Hadoop APIs.

Constructor Summary

Constructors

Constructor	Description
DefaultParquetHandler(FileIO fileIO)	Create an instance of default ParquetHandler implementation.

Method Summary

Modifier and Type	Method	Description
io.delta.kernel.utils.CloseableIterator<io.delta.kernel.engine.FileReadResult>	readParquetFiles(io.delta.kernel.utils.CloseableIterator<io.delta.kernel.utils.FileStatus> fileIter, io.delta.kernel.types.StructType physicalSchema, Optional<io.delta.kernel.expressions.Predicate> predicate)
void	writeParquetFileAtomically(String filePath, io.delta.kernel.utils.CloseableIterator<io.delta.kernel.data.FilteredColumnarBatch> data)	Makes use of LogStore implementations in `delta-storage` to atomically write the data to a file depending upon the destination filesystem.
io.delta.kernel.utils.CloseableIterator<io.delta.kernel.utils.DataFileStatus>	writeParquetFiles(String directoryPath, io.delta.kernel.utils.CloseableIterator<io.delta.kernel.data.FilteredColumnarBatch> dataIter, List<io.delta.kernel.expressions.Column> statsColumns)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

DefaultParquetHandler

public DefaultParquetHandler(FileIO fileIO)

Create an instance of default ParquetHandler implementation.

Parameters:

fileIO - File IO implementation to use for reading and writing files.

Method Details

readParquetFiles

public io.delta.kernel.utils.CloseableIterator<io.delta.kernel.engine.FileReadResult> readParquetFiles(io.delta.kernel.utils.CloseableIterator<io.delta.kernel.utils.FileStatus> fileIter,
 io.delta.kernel.types.StructType physicalSchema,
 Optional<io.delta.kernel.expressions.Predicate> predicate)
                                                                                                throws IOException

Specified by:

readParquetFiles in interface io.delta.kernel.engine.ParquetHandler

Throws:

IOException

writeParquetFiles

public io.delta.kernel.utils.CloseableIterator<io.delta.kernel.utils.DataFileStatus> writeParquetFiles(String directoryPath,
 io.delta.kernel.utils.CloseableIterator<io.delta.kernel.data.FilteredColumnarBatch> dataIter,
 List<io.delta.kernel.expressions.Column> statsColumns)
                                                                                                throws IOException

Specified by:

writeParquetFiles in interface io.delta.kernel.engine.ParquetHandler

Throws:

IOException

writeParquetFileAtomically

public void writeParquetFileAtomically(String filePath,
 io.delta.kernel.utils.CloseableIterator<io.delta.kernel.data.FilteredColumnarBatch> data)
                                throws IOException

Makes use of LogStore implementations in `delta-storage` to atomically write the data to a file depending upon the destination filesystem.

Specified by:

writeParquetFileAtomically in interface io.delta.kernel.engine.ParquetHandler

Parameters:

filePath - Fully qualified destination file path

data - Iterator of FilteredColumnarBatch

Throws:

IOException