Package io.delta.kernel.defaults.engine.hadoopio

Class HadoopFileIO

Object
io.delta.kernel.defaults.engine.hadoopio.HadoopFileIO
All Implemented Interfaces:
FileIO
public class HadoopFileIO extends Object implements FileIO
Implementation of FileIO based on Hadoop APIs.

  • Constructor Details

    • HadoopFileIO

      public HadoopFileIO(org.apache.hadoop.conf.Configuration hadoopConf)

  • Method Details

    • listFrom

      public io.delta.kernel.utils.CloseableIterator<io.delta.kernel.utils.FileStatus> listFrom(String filePath) throws IOException
      Description copied from interface: FileIO
      List the paths in the same directory that are lexicographically greater or equal to (UTF-8 sorting) the given `path`. The result should also be sorted by the file name.
      Specified by:
      listFrom in interface FileIO
      Parameters:
      filePath - Fully qualified path to a file
      Returns:
      Closeable iterator of files. It is the responsibility of the caller to close the iterator.
      Throws:
      IOException - for any other IO error.

    • getFileStatus

      public io.delta.kernel.utils.FileStatus getFileStatus(String path) throws IOException
      Description copied from interface: FileIO
      Get the metadata of the file at the given path.
      Specified by:
      getFileStatus in interface FileIO
      Parameters:
      path - Fully qualified path to the file.
      Returns:
      Metadata of the file.
      Throws:
      IOException - for any IO error.

    • resolvePath

      public String resolvePath(String path) throws IOException
      Description copied from interface: FileIO
      Resolve the given path to a fully qualified path.
      Specified by:
      resolvePath in interface FileIO
      Parameters:
      path - Input path
      Returns:
      Fully qualified path.
      Throws:
      IOException - for any other IO error.

    • mkdirs

      public boolean mkdirs(String path) throws IOException
      Description copied from interface: FileIO
      Create a directory at the given path including parent directories. This mimics the behavior of `mkdir -p` in Unix.
      Specified by:
      mkdirs in interface FileIO
      Parameters:
      path - Full qualified path to create a directory at.
      Returns:
      true if the directory was created successfully, false otherwise.
      Throws:
      IOException - for any IO error.

    • newInputFile

      public InputFile newInputFile(String path, long fileSize)
      Description copied from interface: FileIO
      Get an InputFile for file at given path which can be used to read the file from any arbitrary position in the file.
      Specified by:
      newInputFile in interface FileIO
      Parameters:
      path - Fully qualified path to the file.
      fileSize - Size of the file in bytes.
      Returns:
      InputFile instance.

    • newOutputFile

      public OutputFile newOutputFile(String path)
      Description copied from interface: FileIO
      Create a OutputFile to write new file at the given path.
      Specified by:
      newOutputFile in interface FileIO
      Parameters:
      path - Fully qualified path to the file.
      Returns:
      OutputFile instance which can be used to write to the file.

    • delete

      public boolean delete(String path) throws IOException
      Description copied from interface: FileIO
      Delete the file at given path.
      Specified by:
      delete in interface FileIO
      Parameters:
      path - the path to delete. If path is a directory throws an exception.
      Returns:
      true if delete is successful else false.
      Throws:
      IOException - for any IO error.

    • getConf

      public Optional<String> getConf(String confKey)
      Description copied from interface: FileIO
      Get the configuration value for the given key.

      TODO: should be in a separate interface? may be called ConfigurationProvider?

      Specified by:
      getConf in interface FileIO
      Parameters:
      confKey - configuration key name
      Returns:
      If no such value is present, an Optional.empty() is returned.

    • copyFileAtomically

      public void copyFileAtomically(String srcPath, String destPath, boolean overwrite) throws IOException
      Description copied from interface: FileIO
      Atomically copy a file from source path to destination path. The copy operation should be atomic to ensure that the destination file is either fully copied or not present at all.
      Specified by:
      copyFileAtomically in interface FileIO
      Parameters:
      srcPath - Fully qualified path to the source file to copy
      destPath - Fully qualified path to the destination where the file will be copied
      overwrite - If true, overwrite the destination file if it already exists. If false, throw an exception if the destination exists.
      Throws:
      IOException - for any other IO error