Package org.apache.druid.storage

Interface StorageConnector

All Known Implementing Classes:

ChunkingStorageConnector, LocalFileStorageConnector, NilStorageConnector

public interface StorageConnector

Low level interface for interacting with different storage providers like S3, GCS, Azure and local file system.

For adding a new implementation of this interface in your extension extend StorageConnectorProvider.

For using the interface in your extension as a consumer, use JsonConfigProvider like:

1. JsonConfigProvider.bind(binder, "druid.extension.custom.type", StorageConnectorProvider.class, Custom.class);
2. // bind the storage config provider binder.bind(Key.get(StorageConnector.class, Custom.class)).toProvider(Key.get(StorageConnectorProvider.class, Custom.class)).in(LazySingleton.class);
3. // Use Named annotations to access the storageConnector instance in your custom extension.@Custom StorageConnector storageConnector

For end users, the runtime.properties file would look like

• druid.extension.custom.type="s3"
• druid.extension.custom.bucket="myBucket"

The final state of this interface would have

1. Future Non blocking API's
2. Offset based fetch

Method Summary

Modifier and Type	Method	Description
void	deleteFile(String path)	Delete file present at the input path.
void	deleteFiles(Iterable<String> paths)	Delete files present at the input paths.
void	deleteRecursively(String path)	Delete a directory pointed to by the path and also recursively deletes all files/directories in said directory.
Iterator<String>	listDir(String dirName)	Returns a lazy iterator containing all the files present in the path.
boolean	pathExists(String path)	Check if the path exists in the underlying storage layer.
InputStream	read(String path)	Reads the data present at the path in the underlying storage system.
InputStream	readRange(String path, long from, long size)	Reads the data present for a given range at the path in the underlying storage system.
OutputStream	write(String path)	Open an OutputStream for writing data to the path in the underlying storage system.

Method Details

pathExists

boolean pathExists(String path)
            throws IOException

Check if the path exists in the underlying storage layer. Most implementations prepend the input path with a basePath.

Parameters:

path -

Returns:

true if path exists else false.

Throws:

IOException

read

InputStream read(String path)
          throws IOException

Reads the data present at the path in the underlying storage system. Most implementations prepend the input path with a basePath. The caller should take care of closing the stream when done or in case of error.

Parameters:

path -

Returns:

InputStream

Throws:

IOException - if the path is not present or the unable to read the data present on the path.

readRange

InputStream readRange(String path,
 long from,
 long size)
               throws IOException

Reads the data present for a given range at the path in the underlying storage system. Most implementations prepend the input path with a basePath. The caller should take care of closing the stream when done or in case of error. Further, the caller must ensure that the start offset and the size of the read are valid parameters for the given path for correct behavior.

Parameters:

path - The path to read data from

from - Start offset of the read in the path

size - Length of the read to be done

Returns:

InputStream starting from the given offset limited by the given size

Throws:

IOException - if the path is not present or the unable to read the data present on the path

write

OutputStream write(String path)
            throws IOException

Open an OutputStream for writing data to the path in the underlying storage system. Most implementations prepend the input path with a basePath. If an object exists at the path, the existing object will be overwritten by the write operation. Callers are adivised to namespace there files as there might be race conditions. The caller should take care of closing the stream when done or in case of error.

Parameters:

path - to write

Returns:

OutputStream to the path

Throws:

IOException

deleteFile

void deleteFile(String path)
         throws IOException

Delete file present at the input path. Most implementations prepend the input path with a basePath. If the path is a directory, this method throws an exception.

Parameters:

path - to delete

Throws:

IOException - thrown in case of errors.

deleteFiles

void deleteFiles(Iterable<String> paths)
          throws IOException

Delete files present at the input paths. Most implementations prepend all the input paths with the basePath.
This method is recommended in case we need to delete a batch of files. If the path is a directory, this method throws an exception.

Parameters:

paths - Iterable of the paths to delete.

Throws:

IOException - thrown in case of errors.

deleteRecursively

void deleteRecursively(String path)
                throws IOException

Delete a directory pointed to by the path and also recursively deletes all files/directories in said directory. Most implementations prepend the input path with a basePath.

Parameters:

path - path

Throws:

IOException - thrown in case of errors.

listDir

Iterator<String> listDir(String dirName)
                  throws IOException

Returns a lazy iterator containing all the files present in the path. The returned filenames should be such that joining the dirName and the file name form the full path that can be used as the arguments for other methods of the storage connector. For example, for a S3 path such as s3://bucket/parent1/parent2/child, the filename returned for the input path "parent1/parent2" should be "child" and for input "parent1" should be "parent2/child"

Throws:

IOException