Package com.lancedb.lance.spark.write

Class LanceArrowWriter

java.lang.Object
org.apache.arrow.vector.ipc.ArrowReader
com.lancedb.lance.spark.write.LanceArrowWriter
All Implemented Interfaces:
AutoCloseable, org.apache.arrow.vector.dictionary.DictionaryProvider
public class LanceArrowWriter extends org.apache.arrow.vector.ipc.ArrowReader
A custom arrow reader that supports writes Spark internal rows while reading data in batches.

  • Nested Class Summary

    Nested classes/interfaces inherited from interface org.apache.arrow.vector.dictionary.DictionaryProvider

    org.apache.arrow.vector.dictionary.DictionaryProvider.MapDictionaryProvider

  • Field Summary

    Fields inherited from class org.apache.arrow.vector.ipc.ArrowReader

    allocator, dictionaries

  • Constructor Summary

    Constructors
    Constructor
    Description
    LanceArrowWriter(org.apache.arrow.memory.BufferAllocator allocator, org.apache.arrow.vector.types.pojo.Schema schema, org.apache.spark.sql.types.StructType sparkSchema, int batchSize)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    long
    bytesRead()
     
    protected void
    closeReadSource()
     
    boolean
    loadNextBatch()
     
    void
    prepareLoadNextBatch()
     
    protected org.apache.arrow.vector.types.pojo.Schema
    readSchema()
     

    Methods inherited from class org.apache.arrow.vector.ipc.ArrowReader

    close, close, ensureInitialized, getDictionaryIds, getDictionaryVectors, getVectorSchemaRoot, initialize, loadDictionary, loadRecordBatch, lookup

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • LanceArrowWriter

      public LanceArrowWriter(org.apache.arrow.memory.BufferAllocator allocator, org.apache.arrow.vector.types.pojo.Schema schema, org.apache.spark.sql.types.StructType sparkSchema, int batchSize)

  • Method Details

    • prepareLoadNextBatch

      public void prepareLoadNextBatch() throws IOException
      Overrides:
      prepareLoadNextBatch in class org.apache.arrow.vector.ipc.ArrowReader
      Throws:
      IOException

    • loadNextBatch

      public boolean loadNextBatch() throws IOException
      Specified by:
      loadNextBatch in class org.apache.arrow.vector.ipc.ArrowReader
      Throws:
      IOException

    • bytesRead

      public long bytesRead()
      Specified by:
      bytesRead in class org.apache.arrow.vector.ipc.ArrowReader

    • closeReadSource

      protected void closeReadSource() throws IOException
      Specified by:
      closeReadSource in class org.apache.arrow.vector.ipc.ArrowReader
      Throws:
      IOException

    • readSchema

      protected org.apache.arrow.vector.types.pojo.Schema readSchema()
      Specified by:
      readSchema in class org.apache.arrow.vector.ipc.ArrowReader