@InterfaceAudience.Private @InterfaceStability.Evolving public final class S3AUtils extends Object
| Modifier and Type | Field and Description |
|---|---|
static String |
SSE_C_NO_KEY_ERROR
Encryption SSE-C used but the config lacks an encryption key.
|
static String |
SSE_S3_WITH_KEY_ERROR
Encryption SSE-S3 is used but the caller also set an encryption key.
|
| Modifier and Type | Method and Description |
|---|---|
static void |
clearBucketOption(org.apache.hadoop.conf.Configuration conf,
String bucket,
String genericKey)
Clear a bucket-specific property.
|
static void |
closeAll(org.slf4j.Logger log,
Closeable... closeables)
Close the Closeable objects and ignore any Exception or
null pointers.
|
static AWSCredentialProviderList |
createAWSCredentialProviderSet(URI binding,
org.apache.hadoop.conf.Configuration conf)
Create the AWS credentials from the providers and the URI.
|
static S3AFileStatus |
createFileStatus(org.apache.hadoop.fs.Path keyPath,
com.amazonaws.services.s3.model.S3ObjectSummary summary,
long blockSize,
String owner)
Create a files status instance from a listing.
|
static S3AFileStatus |
createUploadFileStatus(org.apache.hadoop.fs.Path keyPath,
boolean isDir,
long size,
long blockSize,
String owner)
Create a file status for object we just uploaded.
|
static long |
dateToLong(Date date)
Date to long conversion.
|
static int |
ensureOutputParameterInRange(String name,
long size)
Ensure that the long value is in the range of an integer.
|
static IOException |
extractException(String operation,
String path,
ExecutionException ee)
Extract an exception from a failed future, and convert to an IOE.
|
static S3xLoginHelper.Login |
getAWSAccessKeys(URI name,
org.apache.hadoop.conf.Configuration conf)
Return the access key and secret for S3 API use.
|
static String |
getBucketOption(org.apache.hadoop.conf.Configuration conf,
String bucket,
String genericKey)
Get a bucket-specific property.
|
static long |
getMultipartSizeProperty(org.apache.hadoop.conf.Configuration conf,
String property,
long defVal)
Get a size property from the configuration: this property must
be at least equal to
Constants.MULTIPART_MIN_SIZE. |
static boolean |
objectRepresentsDirectory(String name,
long size)
Predicate: does the object represent a directory?.
|
static org.apache.hadoop.conf.Configuration |
propagateBucketOptions(org.apache.hadoop.conf.Configuration source,
String bucket)
Propagates bucket-specific settings into generic S3A configuration keys.
|
static void |
setBucketOption(org.apache.hadoop.conf.Configuration conf,
String bucket,
String genericKey,
String value)
Set a bucket-specific property to a particular value.
|
static String |
stringify(com.amazonaws.services.s3.model.AmazonS3Exception e)
Get low level details of an amazon exception for logging; multi-line.
|
static String |
stringify(com.amazonaws.AmazonServiceException e)
Get low level details of an amazon exception for logging; multi-line.
|
static String |
stringify(com.amazonaws.services.s3.model.S3ObjectSummary summary)
String information about a summary entry for debug messages.
|
static IOException |
translateException(String operation,
org.apache.hadoop.fs.Path path,
com.amazonaws.AmazonClientException exception)
Translate an exception raised in an operation into an IOException.
|
static IOException |
translateException(String operation,
String path,
com.amazonaws.AmazonClientException exception)
Translate an exception raised in an operation into an IOException.
|
public static final String SSE_C_NO_KEY_ERROR
public static final String SSE_S3_WITH_KEY_ERROR
public static IOException translateException(String operation, org.apache.hadoop.fs.Path path, com.amazonaws.AmazonClientException exception)
AmazonClientException passed in, and any status codes included
in the operation. That is: HTTP error codes are examined and can be
used to build a more specific response.operation - operationpath - path operated on (must not be null)exception - amazon exception raisedpublic static IOException translateException(String operation, String path, com.amazonaws.AmazonClientException exception)
AmazonClientException passed in, and any status codes included
in the operation. That is: HTTP error codes are examined and can be
used to build a more specific response.operation - operationpath - path operated on (may be null)exception - amazon exception raisedpublic static IOException extractException(String operation, String path, ExecutionException ee)
operation - operation which failedpath - path operated on (may be null)ee - execution exceptionpublic static String stringify(com.amazonaws.AmazonServiceException e)
e - exceptionpublic static String stringify(com.amazonaws.services.s3.model.AmazonS3Exception e)
e - exceptionpublic static S3AFileStatus createFileStatus(org.apache.hadoop.fs.Path keyPath, com.amazonaws.services.s3.model.S3ObjectSummary summary, long blockSize, String owner)
keyPath - path to entrysummary - summary from AWSblockSize - block size to declare.owner - owner of the filepublic static S3AFileStatus createUploadFileStatus(org.apache.hadoop.fs.Path keyPath, boolean isDir, long size, long blockSize, String owner)
keyPath - path for created objectisDir - true iff directorysize - file lengthblockSize - block size for file statusowner - Hadoop usernamepublic static boolean objectRepresentsDirectory(String name, long size)
name - object namesize - object sizepublic static long dateToLong(Date date)
date - date from AWS querypublic static AWSCredentialProviderList createAWSCredentialProviderSet(URI binding, org.apache.hadoop.conf.Configuration conf) throws IOException
binding - Binding URI, may contain user:pass login detailsconf - filesystem configurationIOException - Problems loading the providers (including reading
secrets from credential files).public static S3xLoginHelper.Login getAWSAccessKeys(URI name, org.apache.hadoop.conf.Configuration conf) throws IOException
name - the URI for which we need the access keys.conf - the Configuration object to interrogate for keys.IOException - problems retrieving passwords from KMS.public static String stringify(com.amazonaws.services.s3.model.S3ObjectSummary summary)
summary - summary objectpublic static long getMultipartSizeProperty(org.apache.hadoop.conf.Configuration conf,
String property,
long defVal)
Constants.MULTIPART_MIN_SIZE.
If it is too small, it is rounded up to that minimum, and a warning
printed.conf - configurationproperty - property namedefVal - default valuepublic static int ensureOutputParameterInRange(String name, long size)
name - property name for error messagessize - original sizepublic static org.apache.hadoop.conf.Configuration propagateBucketOptions(org.apache.hadoop.conf.Configuration source,
String bucket)
fs.s3a.bucket.${bucket}.key to
fs.s3a.key, for all values of "key" other than a small set
of unmodifiable values.
The source of the updated property is set to the key name of the bucket
property, to aid in diagnostics of where things came from.
Returns a new configuration. Why the clone?
You can use the same conf for different filesystems, and the original
values are not updated.
The fs.s3a.impl property cannot be set, nor can
any with the prefix fs.s3a.bucket.
This method does not propagate security provider path information from
the S3A property into the Hadoop common provider: callers must call
patchSecurityCredentialProviders(Configuration) explicitly.source - Source Configuration object.bucket - bucket name. Must not be empty.public static void closeAll(org.slf4j.Logger log,
Closeable... closeables)
IOUtils).log - the log to log at debug level. Can be null.closeables - the objects to closepublic static void setBucketOption(org.apache.hadoop.conf.Configuration conf,
String bucket,
String genericKey,
String value)
fs.s3a. prefix,
that's stripped off, so that when the the bucket properties are propagated
down to the generic values, that value gets copied down.conf - configuration to setbucket - bucket namegenericKey - key; can start with "fs.s3a."value - value to setpublic static void clearBucketOption(org.apache.hadoop.conf.Configuration conf,
String bucket,
String genericKey)
fs.s3a. prefix,
that's stripped off, so that when the the bucket properties are propagated
down to the generic values, that value gets copied down.conf - configuration to setbucket - bucket namegenericKey - key; can start with "fs.s3a."public static String getBucketOption(org.apache.hadoop.conf.Configuration conf, String bucket, String genericKey)
fs.s3a. prefix,
that's stripped off.conf - configuration to setbucket - bucket namegenericKey - key; can start with "fs.s3a."Copyright © 2008–2018 Apache Software Foundation. All rights reserved.