Package org.apache.iceberg.aws.s3

Class S3FileIO

java.lang.Object
org.apache.iceberg.aws.s3.S3FileIO
All Implemented Interfaces:
Closeable, Serializable, AutoCloseable, org.apache.iceberg.io.CredentialSupplier, org.apache.iceberg.io.DelegateFileIO, org.apache.iceberg.io.FileIO, org.apache.iceberg.io.SupportsBulkOperations, org.apache.iceberg.io.SupportsPrefixOperations, org.apache.iceberg.io.SupportsRecoveryOperations, org.apache.iceberg.io.SupportsStorageCredentials
public class S3FileIO extends Object implements org.apache.iceberg.io.CredentialSupplier, org.apache.iceberg.io.DelegateFileIO, org.apache.iceberg.io.SupportsRecoveryOperations, org.apache.iceberg.io.SupportsStorageCredentials
FileIO implementation backed by S3.

Locations used must follow the conventions for S3 URIs (e.g. s3://bucket/path...). URIs with schemes s3a, s3n, https are also treated as s3 file paths. Using this FileIO with other schemes will result in ValidationException.

See Also:

  • Constructor Summary

    Constructors
    Constructor
    Description
    S3FileIO()
    No-arg constructor to load the FileIO dynamically.
    S3FileIO(org.apache.iceberg.util.SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3)
    Constructor with custom s3 supplier and S3FileIO properties.
    S3FileIO(org.apache.iceberg.util.SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3, org.apache.iceberg.util.SerializableSupplier<software.amazon.awssdk.services.s3.S3AsyncClient> s3Async)
    Constructor with custom s3 supplier and s3Async supplier.

  • Method Summary

    Modifier and Type
    Method
    Description
    software.amazon.awssdk.services.s3.S3AsyncClient
    asyncClient()
     
    software.amazon.awssdk.services.s3.S3AsyncClient
    asyncClient(String storagePath)
     
    software.amazon.awssdk.services.s3.S3Client
    client()
     
    software.amazon.awssdk.services.s3.S3Client
    client(String storagePath)
     
    void
    close()
     
    List<org.apache.iceberg.io.StorageCredential>
    credentials()
     
    void
    deleteFile(String path)
     
    void
    deleteFiles(Iterable<String> paths)
    Deletes the given paths in a batched manner.
    void
    deletePrefix(String prefix)
    This method provides a "best-effort" to delete all objects under the given prefix.
    protected void
    finalize()
     
    String
    getCredential()
     
    void
    initialize(Map<String,String> props)
     
    Iterable<org.apache.iceberg.io.FileInfo>
    listPrefix(String prefix)
     
    org.apache.iceberg.io.InputFile
    newInputFile(String path)
     
    org.apache.iceberg.io.InputFile
    newInputFile(String path, long length)
     
    org.apache.iceberg.io.OutputFile
    newOutputFile(String path)
     
    Map<String,String>
    properties()
     
    boolean
    recoverFile(String path)
     
    void
    setCredentials(List<org.apache.iceberg.io.StorageCredential> credentials)
     

    Methods inherited from class java.lang.Object

    clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.iceberg.io.FileIO

    deleteFile, deleteFile, newInputFile, newInputFile, newInputFile, newInputFile

  • Constructor Details

    • S3FileIO

      public S3FileIO()
      No-arg constructor to load the FileIO dynamically.

      All fields are initialized by calling initialize(Map) later.

    • S3FileIO

      public S3FileIO(org.apache.iceberg.util.SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3)
      Constructor with custom s3 supplier and S3FileIO properties.

      Calling initialize(Map) will overwrite information set in this constructor.

      Parameters:
      s3 - s3 supplier

    • S3FileIO

      public S3FileIO(org.apache.iceberg.util.SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3, org.apache.iceberg.util.SerializableSupplier<software.amazon.awssdk.services.s3.S3AsyncClient> s3Async)
      Constructor with custom s3 supplier and s3Async supplier.

      Calling initialize(Map) will overwrite information set in this constructor.

      Parameters:
      s3 - s3 supplier
      s3Async - s3Async supplier

  • Method Details

    • newInputFile

      public org.apache.iceberg.io.InputFile newInputFile(String path)
      Specified by:
      newInputFile in interface org.apache.iceberg.io.FileIO

    • newInputFile

      public org.apache.iceberg.io.InputFile newInputFile(String path, long length)
      Specified by:
      newInputFile in interface org.apache.iceberg.io.FileIO

    • newOutputFile

      public org.apache.iceberg.io.OutputFile newOutputFile(String path)
      Specified by:
      newOutputFile in interface org.apache.iceberg.io.FileIO

    • deleteFile

      public void deleteFile(String path)
      Specified by:
      deleteFile in interface org.apache.iceberg.io.FileIO

    • properties

      public Map<String,String> properties()
      Specified by:
      properties in interface org.apache.iceberg.io.FileIO

    • deleteFiles

      public void deleteFiles(Iterable<String> paths) throws org.apache.iceberg.io.BulkDeletionFailureException
      Deletes the given paths in a batched manner.

      The paths are grouped by bucket, and deletion is triggered when we either reach the configured batch size or have a final remainder batch for each bucket.

      Specified by:
      deleteFiles in interface org.apache.iceberg.io.SupportsBulkOperations
      Parameters:
      paths - paths to delete
      Throws:
      org.apache.iceberg.io.BulkDeletionFailureException

    • listPrefix

      public Iterable<org.apache.iceberg.io.FileInfo> listPrefix(String prefix)
      Specified by:
      listPrefix in interface org.apache.iceberg.io.SupportsPrefixOperations

    • deletePrefix

      public void deletePrefix(String prefix)
      This method provides a "best-effort" to delete all objects under the given prefix.

      Bulk delete operations are used and no reattempt is made for deletes if they fail, but will log any individual objects that are not deleted as part of the bulk operation.

      Specified by:
      deletePrefix in interface org.apache.iceberg.io.SupportsPrefixOperations
      Parameters:
      prefix - prefix to delete

    • client

      public software.amazon.awssdk.services.s3.S3Client client()

    • client

      public software.amazon.awssdk.services.s3.S3Client client(String storagePath)

    • asyncClient

      public software.amazon.awssdk.services.s3.S3AsyncClient asyncClient()

    • asyncClient

      public software.amazon.awssdk.services.s3.S3AsyncClient asyncClient(String storagePath)

    • getCredential

      public String getCredential()
      Specified by:
      getCredential in interface org.apache.iceberg.io.CredentialSupplier

    • initialize

      public void initialize(Map<String,String> props)
      Specified by:
      initialize in interface org.apache.iceberg.io.FileIO

    • close

      public void close()
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Specified by:
      close in interface org.apache.iceberg.io.FileIO

    • finalize

      protected void finalize() throws Throwable
      Overrides:
      finalize in class Object
      Throws:
      Throwable

    • recoverFile

      public boolean recoverFile(String path)
      Specified by:
      recoverFile in interface org.apache.iceberg.io.SupportsRecoveryOperations

    • setCredentials

      public void setCredentials(List<org.apache.iceberg.io.StorageCredential> credentials)
      Specified by:
      setCredentials in interface org.apache.iceberg.io.SupportsStorageCredentials

    • credentials

      public List<org.apache.iceberg.io.StorageCredential> credentials()
      Specified by:
      credentials in interface org.apache.iceberg.io.SupportsStorageCredentials