com.lancedb.lance.spark.write

Class LanceArrowWriter

java.lang.Object

org.apache.arrow.vector.ipc.ArrowReader

com.lancedb.lance.spark.write.LanceArrowWriter

All Implemented Interfaces:

AutoCloseable, org.apache.arrow.vector.dictionary.DictionaryProvider

public class LanceArrowWriter
extends org.apache.arrow.vector.ipc.ArrowReader

A custom arrow reader that supports writes Spark internal rows while reading data in batches.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.arrow.vector.dictionary.DictionaryProvider

org.apache.arrow.vector.dictionary.DictionaryProvider.MapDictionaryProvider

Field Summary

Fields inherited from class org.apache.arrow.vector.ipc.ArrowReader

allocator, dictionaries

Constructor Summary

Constructors

Constructor and Description
LanceArrowWriter(org.apache.arrow.memory.BufferAllocator allocator, org.apache.arrow.vector.types.pojo.Schema schema, int batchSize)

Method Summary

Modifier and Type	Method and Description
long	bytesRead()
protected void	closeReadSource()
boolean	loadNextBatch()
void	prepareLoadNextBatch()
protected org.apache.arrow.vector.types.pojo.Schema	readSchema()

Methods inherited from class org.apache.arrow.vector.ipc.ArrowReader

close, close, ensureInitialized, getDictionaryIds, getDictionaryVectors, getVectorSchemaRoot, initialize, loadDictionary, loadRecordBatch, lookup

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LanceArrowWriter

public LanceArrowWriter(org.apache.arrow.memory.BufferAllocator allocator,
                        org.apache.arrow.vector.types.pojo.Schema schema,
                        int batchSize)

Method Detail

prepareLoadNextBatch

public void prepareLoadNextBatch()
                          throws IOException

Overrides:

prepareLoadNextBatch in class org.apache.arrow.vector.ipc.ArrowReader

Throws:

IOException

loadNextBatch

public boolean loadNextBatch()
                      throws IOException

Specified by:

loadNextBatch in class org.apache.arrow.vector.ipc.ArrowReader

Throws:

IOException

bytesRead

public long bytesRead()

Specified by:

bytesRead in class org.apache.arrow.vector.ipc.ArrowReader

closeReadSource

protected void closeReadSource()
                        throws IOException

Specified by:

closeReadSource in class org.apache.arrow.vector.ipc.ArrowReader

Throws:

IOException

readSchema

protected org.apache.arrow.vector.types.pojo.Schema readSchema()

Specified by:

readSchema in class org.apache.arrow.vector.ipc.ArrowReader