Package io.delta.kernel.defaults.engine.fileio

Interface FileIO

All Known Implementing Classes:
HadoopFileIO
public interface FileIO
Interface for file IO operations. Connectors can implement their own version of the FileIO depending upon their environment. The DefaultEngine takes FileIO instance as input and all I/O operations from the default engine are done using the passed in FileIO instance.

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    copyFileAtomically(String srcPath, String destPath, boolean overwrite)
    Atomically copy a file from source path to destination path.
    boolean
    delete(String path)
    Delete the file at given path.
    Optional<String>
    getConf(String confKey)
    Get the configuration value for the given key.
    io.delta.kernel.utils.FileStatus
    getFileStatus(String path)
    Get the metadata of the file at the given path.
    io.delta.kernel.utils.CloseableIterator<io.delta.kernel.utils.FileStatus>
    listFrom(String filePath)
    List the paths in the same directory that are lexicographically greater or equal to (UTF-8 sorting) the given `path`.
    boolean
    mkdirs(String path)
    Create a directory at the given path including parent directories.
    InputFile
    newInputFile(String path, long fileSize)
    Get an InputFile for file at given path which can be used to read the file from any arbitrary position in the file.
    OutputFile
    newOutputFile(String path)
    Create a OutputFile to write new file at the given path.
    String
    resolvePath(String path)
    Resolve the given path to a fully qualified path.

  • Method Details

    • listFrom

      io.delta.kernel.utils.CloseableIterator<io.delta.kernel.utils.FileStatus> listFrom(String filePath) throws IOException
      List the paths in the same directory that are lexicographically greater or equal to (UTF-8 sorting) the given `path`. The result should also be sorted by the file name.
      Parameters:
      filePath - Fully qualified path to a file
      Returns:
      Closeable iterator of files. It is the responsibility of the caller to close the iterator.
      Throws:
      FileNotFoundException - if the file at the given path is not found
      IOException - for any other IO error.

    • getFileStatus

      io.delta.kernel.utils.FileStatus getFileStatus(String path) throws IOException
      Get the metadata of the file at the given path.
      Parameters:
      path - Fully qualified path to the file.
      Returns:
      Metadata of the file.
      Throws:
      IOException - for any IO error.

    • resolvePath

      String resolvePath(String path) throws IOException
      Resolve the given path to a fully qualified path.
      Parameters:
      path - Input path
      Returns:
      Fully qualified path.
      Throws:
      FileNotFoundException - If the given path doesn't exist.
      IOException - for any other IO error.

    • mkdirs

      boolean mkdirs(String path) throws IOException
      Create a directory at the given path including parent directories. This mimics the behavior of `mkdir -p` in Unix.
      Parameters:
      path - Full qualified path to create a directory at.
      Returns:
      true if the directory was created successfully, false otherwise.
      Throws:
      IOException - for any IO error.

    • newInputFile

      InputFile newInputFile(String path, long fileSize)
      Get an InputFile for file at given path which can be used to read the file from any arbitrary position in the file.
      Parameters:
      path - Fully qualified path to the file.
      fileSize - Size of the file in bytes.
      Returns:
      InputFile instance.

    • newOutputFile

      OutputFile newOutputFile(String path)
      Create a OutputFile to write new file at the given path.
      Parameters:
      path - Fully qualified path to the file.
      Returns:
      OutputFile instance which can be used to write to the file.

    • delete

      boolean delete(String path) throws IOException
      Delete the file at given path.
      Parameters:
      path - the path to delete. If path is a directory throws an exception.
      Returns:
      true if delete is successful else false.
      Throws:
      IOException - for any IO error.

    • getConf

      Optional<String> getConf(String confKey)
      Get the configuration value for the given key.

      TODO: should be in a separate interface? may be called ConfigurationProvider?

      Parameters:
      confKey - configuration key name
      Returns:
      If no such value is present, an Optional.empty() is returned.

    • copyFileAtomically

      void copyFileAtomically(String srcPath, String destPath, boolean overwrite) throws IOException
      Atomically copy a file from source path to destination path. The copy operation should be atomic to ensure that the destination file is either fully copied or not present at all.
      Parameters:
      srcPath - Fully qualified path to the source file to copy
      destPath - Fully qualified path to the destination where the file will be copied
      overwrite - If true, overwrite the destination file if it already exists. If false, throw an exception if the destination exists.
      Throws:
      FileAlreadyExistsException - if the destination file already exists and overwrite is false.
      FileNotFoundException - if the source file does not exist
      IOException - for any other IO error