Package io.delta.kernel.defaults.engine.hadoopio

Class HadoopOutputFile

Object
io.delta.kernel.defaults.engine.hadoopio.HadoopOutputFile
All Implemented Interfaces:
OutputFile
public class HadoopOutputFile extends Object implements OutputFile

  • Constructor Details

    • HadoopOutputFile

      public HadoopOutputFile(org.apache.hadoop.conf.Configuration hadoopConf, String path)

  • Method Details

    • path

      public String path()
      Description copied from interface: OutputFile
      Get the path of the file.
      Specified by:
      path in interface OutputFile
      Returns:
      the path of the file.

    • create

      public PositionOutputStream create(boolean putIfAbsent) throws IOException
      Description copied from interface: OutputFile
      Get the output stream to write to the file.
      • If the file already exists, (either at the time of creating the PositionOutputStream or at the time of closing it), it will be overwritten.
      • if atomicWrite is set, then the entire content is written or none, but won't create a file with the partial contents.
      Specified by:
      create in interface OutputFile
      Returns:
      the output stream to write to the file. It is the responsibility of the caller to close the stream.
      Throws:
      IOException - if an I/O error occurs.

    • writeAtomically

      public void writeAtomically(io.delta.kernel.utils.CloseableIterator<String> data, boolean overwrite) throws IOException
      Description copied from interface: OutputFile
      Atomically write (either write is completely or don't write all - i.e. don't leave file with partial content) the data to a file at the given path. If the file already exists do not replace it if replace is false. If replace is true, then replace the file with the new data.

      TODO: the semantics are very loose here, see if there is a better API name. One of the reasons why the data is passed as an iterator is because of the existing LogStore interface which are used in the Hadoop implementation of the FileIO

      Specified by:
      writeAtomically in interface OutputFile
      Parameters:
      data - the data to write. Each element in the iterator is a line in the file.
      overwrite - if true, overwrite the file with the new data. If false, do not overwrite the file.
      Throws:
      IOException - for any IO error.