Package io.delta.kernel.defaults.engine

Class DefaultFileSystemClient

Object

io.delta.kernel.defaults.engine.DefaultFileSystemClient

All Implemented Interfaces:

io.delta.kernel.engine.FileSystemClient

public class DefaultFileSystemClient
extends Object
implements io.delta.kernel.engine.FileSystemClient

Default implementation of FileSystemClient based on Hadoop APIs. It takes a Hadoop Configuration object to interact with the file system. The following optional configurations can be set to customize the behavior of the client:

• io.delta.kernel.logStore.<scheme>.impl - The class name of the custom LogStore implementation to use for operations on storage systems with the specified scheme. For example, to use a custom LogStore for S3 storage objects:

  
   <property>
     <name>io.delta.kernel.logStore.s3.impl</name>
     <value>com.example.S3LogStore</value>
   </property>
  
   

  If not set, the default LogStore implementation for the scheme will be used.
• delta.enableFastS3AListFrom - Set to true to enable fast listing functionality when using a LogStore created for S3 storage objects.

The above list of options is not exhaustive. For a complete list of options, refer to the specific implementation of FileSystem.

Constructor Summary

Constructors

Constructor	Description
DefaultFileSystemClient(FileIO fileIO)	Create an instance of the default FileSystemClient implementation.

Method Summary

Modifier and Type	Method	Description
void	copyFileAtomically(String srcPath, String destPath, boolean overwrite)
boolean	delete(String path)
io.delta.kernel.utils.FileStatus	getFileStatus(String path)
io.delta.kernel.utils.CloseableIterator<io.delta.kernel.utils.FileStatus>	listFrom(String filePath)
boolean	mkdirs(String path)
io.delta.kernel.utils.CloseableIterator<ByteArrayInputStream>	readFiles(io.delta.kernel.utils.CloseableIterator<io.delta.kernel.engine.FileReadRequest> readRequests)
String	resolvePath(String path)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

DefaultFileSystemClient

public DefaultFileSystemClient(FileIO fileIO)

Create an instance of the default FileSystemClient implementation.

Parameters:

fileIO - The FileIO implementation to use for file operations.

Method Details

listFrom

public io.delta.kernel.utils.CloseableIterator<io.delta.kernel.utils.FileStatus> listFrom(String filePath)
                                                                                   throws IOException

Specified by:

listFrom in interface io.delta.kernel.engine.FileSystemClient

Throws:

IOException

resolvePath

public String resolvePath(String path)
                   throws IOException

Specified by:

resolvePath in interface io.delta.kernel.engine.FileSystemClient

Throws:

IOException

readFiles

public io.delta.kernel.utils.CloseableIterator<ByteArrayInputStream> readFiles(io.delta.kernel.utils.CloseableIterator<io.delta.kernel.engine.FileReadRequest> readRequests)
                                                                        throws IOException

Specified by:

readFiles in interface io.delta.kernel.engine.FileSystemClient

Throws:

IOException

mkdirs

public boolean mkdirs(String path)
               throws IOException

Specified by:

mkdirs in interface io.delta.kernel.engine.FileSystemClient

Throws:

IOException

delete

public boolean delete(String path)
               throws IOException

Specified by:

delete in interface io.delta.kernel.engine.FileSystemClient

Throws:

IOException

getFileStatus

public io.delta.kernel.utils.FileStatus getFileStatus(String path)
                                               throws IOException

Specified by:

getFileStatus in interface io.delta.kernel.engine.FileSystemClient

Throws:

IOException

copyFileAtomically

public void copyFileAtomically(String srcPath,
 String destPath,
 boolean overwrite)
                        throws IOException

Specified by:

copyFileAtomically in interface io.delta.kernel.engine.FileSystemClient

Throws:

IOException