@InterfaceAudience.Private @InterfaceStability.Evolving public final class S3AUtils extends Object
| Modifier and Type | Method and Description |
|---|---|
static void |
closeAll(org.slf4j.Logger log,
Closeable... closeables)
Close the Closeable objects and ignore any Exception or
null pointers.
|
static AWSCredentialProviderList |
createAWSCredentialProviderSet(URI binding,
org.apache.hadoop.conf.Configuration conf,
URI fsURI)
Create the AWS credentials from the providers and the URI.
|
static S3AFileStatus |
createFileStatus(org.apache.hadoop.fs.Path keyPath,
com.amazonaws.services.s3.model.S3ObjectSummary summary,
long blockSize,
String owner)
Create a files status instance from a listing.
|
static long |
dateToLong(Date date)
Date to long conversion.
|
static int |
ensureOutputParameterInRange(String name,
long size)
Ensure that the long value is in the range of an integer.
|
static IOException |
extractException(String operation,
String path,
ExecutionException ee)
Extract an exception from a failed future, and convert to an IOE.
|
static S3xLoginHelper.Login |
getAWSAccessKeys(URI name,
org.apache.hadoop.conf.Configuration conf)
Return the access key and secret for S3 API use.
|
static long |
getMultipartSizeProperty(org.apache.hadoop.conf.Configuration conf,
String property,
long defVal)
Get a size property from the configuration: this property must
be at least equal to
Constants.MULTIPART_MIN_SIZE. |
static boolean |
objectRepresentsDirectory(String name,
long size)
Predicate: does the object represent a directory?.
|
static org.apache.hadoop.conf.Configuration |
propagateBucketOptions(org.apache.hadoop.conf.Configuration source,
String bucket)
Propagates bucket-specific settings into generic S3A configuration keys.
|
static String |
stringify(com.amazonaws.services.s3.model.AmazonS3Exception e)
Get low level details of an amazon exception for logging; multi-line.
|
static String |
stringify(com.amazonaws.AmazonServiceException e)
Get low level details of an amazon exception for logging; multi-line.
|
static String |
stringify(com.amazonaws.services.s3.model.S3ObjectSummary summary)
String information about a summary entry for debug messages.
|
static IOException |
translateException(String operation,
org.apache.hadoop.fs.Path path,
com.amazonaws.AmazonClientException exception)
Translate an exception raised in an operation into an IOException.
|
static IOException |
translateException(String operation,
String path,
com.amazonaws.AmazonClientException exception)
Translate an exception raised in an operation into an IOException.
|
public static IOException translateException(String operation, org.apache.hadoop.fs.Path path, com.amazonaws.AmazonClientException exception)
AmazonClientException passed in, and any status codes included
in the operation. That is: HTTP error codes are examined and can be
used to build a more specific response.operation - operationpath - path operated on (must not be null)exception - amazon exception raisedpublic static IOException translateException(String operation, String path, com.amazonaws.AmazonClientException exception)
AmazonClientException passed in, and any status codes included
in the operation. That is: HTTP error codes are examined and can be
used to build a more specific response.operation - operationpath - path operated on (may be null)exception - amazon exception raisedpublic static IOException extractException(String operation, String path, ExecutionException ee)
operation - operation which failedpath - path operated on (may be null)ee - execution exceptionpublic static String stringify(com.amazonaws.AmazonServiceException e)
e - exceptionpublic static String stringify(com.amazonaws.services.s3.model.AmazonS3Exception e)
e - exceptionpublic static S3AFileStatus createFileStatus(org.apache.hadoop.fs.Path keyPath, com.amazonaws.services.s3.model.S3ObjectSummary summary, long blockSize, String owner)
keyPath - path to entrysummary - summary from AWSblockSize - block size to declare.owner - owner of the filepublic static boolean objectRepresentsDirectory(String name, long size)
name - object namesize - object sizepublic static long dateToLong(Date date)
date - date from AWS querypublic static AWSCredentialProviderList createAWSCredentialProviderSet(URI binding, org.apache.hadoop.conf.Configuration conf, URI fsURI) throws IOException
binding - Binding URI, may contain user:pass login detailsconf - filesystem configurationfsURI - fS URI —after any login details have been stripped.IOException - Problems loading the providers (including reading
secrets from credential files).public static S3xLoginHelper.Login getAWSAccessKeys(URI name, org.apache.hadoop.conf.Configuration conf) throws IOException
name - the URI for which we need the access keys.conf - the Configuration object to interrogate for keys.IOException - problems retrieving passwords from KMS.public static String stringify(com.amazonaws.services.s3.model.S3ObjectSummary summary)
summary - summary objectpublic static long getMultipartSizeProperty(org.apache.hadoop.conf.Configuration conf,
String property,
long defVal)
Constants.MULTIPART_MIN_SIZE.
If it is too small, it is rounded up to that minimum, and a warning
printed.conf - configurationproperty - property namedefVal - default valuepublic static int ensureOutputParameterInRange(String name, long size)
name - property name for error messagessize - original sizepublic static org.apache.hadoop.conf.Configuration propagateBucketOptions(org.apache.hadoop.conf.Configuration source,
String bucket)
fs.s3a.bucket.${bucket}.key to
fs.s3a.key, for all values of "key" other than a small set
of unmodifiable values.
The source of the updated property is set to the key name of the bucket
property, to aid in diagnostics of where things came from.
Returns a new configuration. Why the clone?
You can use the same conf for different filesystems, and the original
values are not updated.
The fs.s3a.impl property cannot be set, nor can
any with the prefix fs.s3a.bucket.
This method does not propagate security provider path information from
the S3A property into the Hadoop common provider: callers must call
patchSecurityCredentialProviders(Configuration) explicitly.source - Source Configuration object.bucket - bucket name. Must not be empty.public static void closeAll(org.slf4j.Logger log,
Closeable... closeables)
IOUtils).log - the log to log at debug level. Can be null.closeables - the objects to closeCopyright © 2017 Apache Software Foundation. All Rights Reserved.