Package io.delta.kernel.defaults.internal.parquet

Class ParquetFileWriter

Object

io.delta.kernel.defaults.internal.parquet.ParquetFileWriter

public class ParquetFileWriter
extends Object

Implements writing data given as FilteredColumnarBatch to Parquet files.

It makes use of the `parquet-mr` library to write the data in Parquet format. The main class used is ParquetWriter which is used to write the data row by row to the Parquet file. Supporting interface for this writer is WriteSupport (in this writer implementation, it is ParquetFileWriter.BatchWriteSupport). ParquetFileWriter.BatchWriteSupport, on call back from ParquetWriter, reads the contents of ColumnarBatch and passes the contents to ParquetWriter through RecordConsumer.

Field Summary

Fields

Modifier and Type	Field	Description
static final long	DEFAULT_TARGET_FILE_SIZE
static final String	TARGET_FILE_SIZE_CONF

Method Summary

Modifier and Type	Method	Description
static ParquetFileWriter	multiFileWriter(FileIO fileIO, String location, List<io.delta.kernel.expressions.Column> statsColumns)	Create writer to write data into one or more files depending upon the delta.kernel.default.parquet.writer.targetMaxFileSize value and the given data.
static ParquetFileWriter	singleFileWriter(FileIO fileIO, String location, boolean atomicWrite, List<io.delta.kernel.expressions.Column> statsColumns)	Create writer to write the data exactly into one file.
io.delta.kernel.utils.CloseableIterator<io.delta.kernel.utils.DataFileStatus>	write(io.delta.kernel.utils.CloseableIterator<io.delta.kernel.data.FilteredColumnarBatch> dataIter)	Write the given data to Parquet files.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

TARGET_FILE_SIZE_CONF

public static final String TARGET_FILE_SIZE_CONF

See Also:

• Constant Field Values

DEFAULT_TARGET_FILE_SIZE

public static final long DEFAULT_TARGET_FILE_SIZE

See Also:

• Constant Field Values

Method Details

multiFileWriter

public static ParquetFileWriter multiFileWriter(FileIO fileIO,
 String location,
 List<io.delta.kernel.expressions.Column> statsColumns)

Create writer to write data into one or more files depending upon the delta.kernel.default.parquet.writer.targetMaxFileSize value and the given data.

Parameters:

fileIO - File IO implementation to use for reading and writing files.

location - Location to write the data. Should be a directory.

statsColumns - List of columns to collect statistics for. The statistics collection is optional.

singleFileWriter

public static ParquetFileWriter singleFileWriter(FileIO fileIO,
 String location,
 boolean atomicWrite,
 List<io.delta.kernel.expressions.Column> statsColumns)

Create writer to write the data exactly into one file.

Parameters:

fileIO - File IO implementation to use for reading and writing files.

location - Location to write the data. Shouldn't be a directory.

atomicWrite - If true, write the file is written atomically (i.e. either the entire content is written or none, but won't create a file with the partial contents).

statsColumns - List of columns to collect statistics for. The statistics collection is optional.

write

public io.delta.kernel.utils.CloseableIterator<io.delta.kernel.utils.DataFileStatus> write(io.delta.kernel.utils.CloseableIterator<io.delta.kernel.data.FilteredColumnarBatch> dataIter)

Write the given data to Parquet files.

Parameters:

dataIter - Iterator of data to write.

Returns:

an iterator of DataFileStatus where each entry contains the metadata of the data file written. It is the responsibility of the caller to close the iterator.