@InterfaceAudience.Private
@InterfaceStability.Evolving
public class S3AFileSystem
extends org.apache.hadoop.fs.FileSystem
FileSystem.get(Configuration) and variants to
create one.
If cast to S3AFileSystem, extra methods and features may be accessed.
Consider those private and unstable.
Because it prints some of the state of the instrumentation,
the output of toString() must also be considered unstable.| Modifier and Type | Field and Description |
|---|---|
static int |
DEFAULT_BLOCKSIZE
Default blocksize as used in blocksize and FS status queries.
|
static org.slf4j.Logger |
LOG |
| Constructor and Description |
|---|
S3AFileSystem() |
| Modifier and Type | Method and Description |
|---|---|
org.apache.hadoop.fs.FSDataOutputStream |
append(org.apache.hadoop.fs.Path f,
int bufferSize,
org.apache.hadoop.util.Progressable progress)
Append to an existing file (optional operation).
|
protected URI |
canonicalizeUri(URI rawUri) |
void |
checkPath(org.apache.hadoop.fs.Path path)
Check that a Path belongs to this FileSystem.
|
void |
close()
Close the filesystem.
|
protected S3ListResult |
continueListObjects(S3ListRequest request,
S3ListResult prevResult)
List the next set of objects.
|
void |
copyFromLocalFile(boolean delSrc,
boolean overwrite,
org.apache.hadoop.fs.Path src,
org.apache.hadoop.fs.Path dst)
The src file is on the local disk.
|
org.apache.hadoop.fs.FSDataOutputStream |
create(org.apache.hadoop.fs.Path f,
org.apache.hadoop.fs.permission.FsPermission permission,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
org.apache.hadoop.util.Progressable progress)
Create an FSDataOutputStream at the indicated Path with write-progress
reporting.
|
org.apache.hadoop.fs.FSDataOutputStream |
createNonRecursive(org.apache.hadoop.fs.Path path,
org.apache.hadoop.fs.permission.FsPermission permission,
EnumSet<org.apache.hadoop.fs.CreateFlag> flags,
int bufferSize,
short replication,
long blockSize,
org.apache.hadoop.util.Progressable progress) |
protected void |
decrementGauge(Statistic statistic,
long count)
Decrement a gauge by a specific value.
|
boolean |
delete(org.apache.hadoop.fs.Path f,
boolean recursive)
Delete a Path.
|
boolean |
exists(org.apache.hadoop.fs.Path f)
Override superclass so as to add statistic collection.
|
String |
getBucket()
Get the bucket of this filesystem.
|
String |
getBucketLocation()
Get the region of a bucket.
|
String |
getBucketLocation(String bucketName)
Get the region of a bucket.
|
String |
getCanonicalServiceName()
Override getCanonicalServiceName because we don't support token in S3A.
|
long |
getDefaultBlockSize()
Deprecated.
use
FileSystem.getDefaultBlockSize(Path) instead |
int |
getDefaultPort() |
org.apache.hadoop.fs.FileStatus |
getFileStatus(org.apache.hadoop.fs.Path f)
Return a file status object that represents the path.
|
S3AInputPolicy |
getInputPolicy()
Get the input policy for this FS instance.
|
S3AInstrumentation |
getInstrumentation()
Get S3A Instrumentation.
|
MetadataStore |
getMetadataStore()
Get the metadata store.
|
long |
getMultiPartThreshold()
Get the threshold for multipart files.
|
com.amazonaws.services.s3.model.ObjectMetadata |
getObjectMetadata(org.apache.hadoop.fs.Path path)
Low-level call to get at the object metadata.
|
protected com.amazonaws.services.s3.model.ObjectMetadata |
getObjectMetadata(String key)
Request object metadata; increments counters in the process.
|
long |
getPartitionSize()
Get the partition size for multipart operations.
|
String |
getScheme()
Return the protocol scheme for the FileSystem.
|
S3AStorageStatistics |
getStorageStatistics()
Get the storage statistics of this filesystem.
|
URI |
getUri()
Returns a URI whose scheme and authority identify this FileSystem.
|
String |
getUsername()
Get the username of the FS.
|
org.apache.hadoop.fs.Path |
getWorkingDirectory()
Get the current working directory for the given file system.
|
org.apache.hadoop.fs.FileStatus[] |
globStatus(org.apache.hadoop.fs.Path pathPattern)
Increments the statistic
Statistic.INVOCATION_GLOB_STATUS. |
org.apache.hadoop.fs.FileStatus[] |
globStatus(org.apache.hadoop.fs.Path pathPattern,
org.apache.hadoop.fs.PathFilter filter)
Override superclass so as to add statistic collection.
|
boolean |
hasMetadataStore()
Does this Filesystem have a metadata store?
|
protected void |
incrementGauge(Statistic statistic,
long count)
Increment a gauge by a specific value.
|
void |
incrementPutCompletedStatistics(boolean success,
long bytes)
At the end of a put/multipart upload operation, update the
relevant counters and gauges.
|
void |
incrementPutProgressStatistics(String key,
long bytes)
Callback for use in progress callbacks from put/multipart upload events.
|
void |
incrementPutStartStatistics(long bytes)
At the start of a put/multipart upload operation, update the
relevant counters.
|
void |
incrementReadOperations()
Increment read operations.
|
protected void |
incrementStatistic(Statistic statistic)
Increment a statistic by 1.
|
protected void |
incrementStatistic(Statistic statistic,
long count)
Increment a statistic by a specific value.
|
void |
incrementWriteOperations()
Increment the write operation counter.
|
void |
initialize(URI name,
org.apache.hadoop.conf.Configuration originalConf)
Called after a new FileSystem instance is constructed.
|
org.apache.hadoop.fs.FileStatus[] |
innerListStatus(org.apache.hadoop.fs.Path f)
List the statuses of the files/directories in the given path if the path is
a directory.
|
boolean |
isDirectory(org.apache.hadoop.fs.Path f)
Override superclass so as to add statistic collection.
|
boolean |
isFile(org.apache.hadoop.fs.Path f)
Override superclass so as to add statistic collection.
|
org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> |
listFiles(org.apache.hadoop.fs.Path f,
boolean recursive)
.
|
org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> |
listFilesAndEmptyDirectories(org.apache.hadoop.fs.Path f,
boolean recursive) |
org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> |
listLocatedStatus(org.apache.hadoop.fs.Path f)
Override superclass so as to add statistic collection.
|
org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> |
listLocatedStatus(org.apache.hadoop.fs.Path f,
org.apache.hadoop.fs.PathFilter filter)
.
|
protected S3ListResult |
listObjects(S3ListRequest request)
Initiate a
listObjects operation, incrementing metrics
in the process. |
org.apache.hadoop.fs.FileStatus[] |
listStatus(org.apache.hadoop.fs.Path f)
List the statuses of the files/directories in the given path if the path is
a directory.
|
boolean |
mkdirs(org.apache.hadoop.fs.Path path,
org.apache.hadoop.fs.permission.FsPermission permission)
Make the given path and all non-existent parents into
directories.
|
com.amazonaws.services.s3.model.ObjectMetadata |
newObjectMetadata()
Create a new object metadata instance.
|
com.amazonaws.services.s3.model.ObjectMetadata |
newObjectMetadata(long length)
Create a new object metadata instance.
|
com.amazonaws.services.s3.model.PutObjectRequest |
newPutObjectRequest(String key,
com.amazonaws.services.s3.model.ObjectMetadata metadata,
File srcfile)
Create a putObject request.
|
org.apache.hadoop.fs.FSDataInputStream |
open(org.apache.hadoop.fs.Path f,
int bufferSize)
Opens an FSDataInputStream at the indicated Path.
|
String |
pathToKey(org.apache.hadoop.fs.Path path)
Turns a path (relative or otherwise) into an S3 key.
|
UploadInfo |
putObject(com.amazonaws.services.s3.model.PutObjectRequest putObjectRequest)
Start a transfer-manager managed async PUT of an object,
incrementing the put requests and put bytes
counters.
|
org.apache.hadoop.fs.Path |
qualify(org.apache.hadoop.fs.Path path)
Qualify a path.
|
boolean |
rename(org.apache.hadoop.fs.Path src,
org.apache.hadoop.fs.Path dst)
Renames Path src to Path dst.
|
void |
setInputPolicy(S3AInputPolicy inputPolicy)
Change the input policy for this FS.
|
protected void |
setOptionalCopyObjectRequestParameters(com.amazonaws.services.s3.model.CopyObjectRequest copyObjectRequest) |
protected void |
setOptionalMultipartUploadRequestParameters(com.amazonaws.services.s3.model.InitiateMultipartUploadRequest req) |
protected void |
setOptionalUploadPartRequestParameters(com.amazonaws.services.s3.model.UploadPartRequest request)
Sets server side encryption parameters to the part upload
request when encryption is enabled.
|
void |
setWorkingDirectory(org.apache.hadoop.fs.Path newDir)
Set the current working directory for the given file system.
|
String |
toString() |
com.amazonaws.services.s3.model.UploadPartResult |
uploadPart(com.amazonaws.services.s3.model.UploadPartRequest request)
Upload part of a multi-partition file.
|
protected void |
verifyBucketExists()
Verify that the bucket exists.
|
access, addDelegationTokens, append, append, appendFile, areSymlinksEnabled, cancelDeleteOnExit, clearStatistics, closeAll, closeAllForUGI, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createFile, createNewFile, createNonRecursive, createNonRecursive, createSnapshot, createSnapshot, createSymlink, delete, deleteOnExit, deleteSnapshot, enableSymlinks, fixRelativePart, get, get, get, getAclStatus, getAllStatistics, getAllStoragePolicies, getBlockSize, getCanonicalUri, getChildFileSystems, getContentSummary, getDefaultBlockSize, getDefaultReplication, getDefaultReplication, getDefaultUri, getDelegationToken, getFileBlockLocations, getFileBlockLocations, getFileChecksum, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getHomeDirectory, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getQuotaUsage, getReplication, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getStoragePolicy, getTrashRoot, getTrashRoots, getUsed, getUsed, getXAttr, getXAttrs, getXAttrs, listCorruptFileBlocks, listStatus, listStatus, listStatus, listStatusBatch, listStatusIterator, listXAttrs, makeQualified, mkdirs, mkdirs, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, newInstance, newInstance, newInstance, newInstanceLocal, open, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameSnapshot, resolveLink, resolvePath, setAcl, setDefaultUri, setDefaultUri, setOwner, setPermission, setReplication, setStoragePolicy, setTimes, setVerifyChecksum, setWriteChecksum, setXAttr, setXAttr, startLocalOutput, supportsSymlinks, truncate, unsetStoragePolicypublic static final int DEFAULT_BLOCKSIZE
public static final org.slf4j.Logger LOG
public void initialize(URI name, org.apache.hadoop.conf.Configuration originalConf) throws IOException
initialize in class org.apache.hadoop.fs.FileSystemname - a uri whose authority section names the host, port, etc.
for this FileSystemoriginalConf - the configuration to use for the FS. The
bucket-specific options are patched over the base ones before any use is
made of the config.IOExceptionprotected void verifyBucketExists()
throws FileNotFoundException,
IOException
FileNotFoundException - the bucket is absentIOException - any other problem talking to S3public S3AInstrumentation getInstrumentation()
public String getScheme()
getScheme in class org.apache.hadoop.fs.FileSystempublic URI getUri()
getUri in class org.apache.hadoop.fs.FileSystempublic int getDefaultPort()
getDefaultPort in class org.apache.hadoop.fs.FileSystempublic String getBucketLocation() throws IOException
IOException - on any failure.public String getBucketLocation(String bucketName) throws IOException
bucketName - the name of the bucketIOException - on any failure.@InterfaceStability.Unstable public S3AInputPolicy getInputPolicy()
public String getBucket()
@InterfaceStability.Unstable public void setInputPolicy(S3AInputPolicy inputPolicy)
inputPolicy - new policypublic String pathToKey(org.apache.hadoop.fs.Path path)
path - input path, may be relative to the working dirpublic org.apache.hadoop.fs.Path qualify(org.apache.hadoop.fs.Path path)
path - path to qualifypublic void checkPath(org.apache.hadoop.fs.Path path)
checkPath in class org.apache.hadoop.fs.FileSystempath - to checkIllegalArgumentException - if there is an FS mismatchprotected URI canonicalizeUri(URI rawUri)
canonicalizeUri in class org.apache.hadoop.fs.FileSystempublic org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path f,
int bufferSize)
throws IOException
open in class org.apache.hadoop.fs.FileSystemf - the file name to openbufferSize - the size of the buffer to be used.IOExceptionpublic org.apache.hadoop.fs.FSDataOutputStream create(org.apache.hadoop.fs.Path f,
org.apache.hadoop.fs.permission.FsPermission permission,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
org.apache.hadoop.util.Progressable progress)
throws IOException
create in class org.apache.hadoop.fs.FileSystemf - the file name to openpermission - the permission to set.overwrite - if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize - the size of the buffer to be used.replication - required block replication for the file.blockSize - the requested block size.progress - the progress reporter.IOException - in the event of IO related errors.FileSystem.setPermission(Path, FsPermission)public org.apache.hadoop.fs.FSDataOutputStream createNonRecursive(org.apache.hadoop.fs.Path path,
org.apache.hadoop.fs.permission.FsPermission permission,
EnumSet<org.apache.hadoop.fs.CreateFlag> flags,
int bufferSize,
short replication,
long blockSize,
org.apache.hadoop.util.Progressable progress)
throws IOException
createNonRecursive in class org.apache.hadoop.fs.FileSystemFileNotFoundException - if the parent directory is not present -or
is not a directory.IOExceptionpublic org.apache.hadoop.fs.FSDataOutputStream append(org.apache.hadoop.fs.Path f,
int bufferSize,
org.apache.hadoop.util.Progressable progress)
throws IOException
append in class org.apache.hadoop.fs.FileSystemf - the existing file to be appended.bufferSize - the size of the buffer to be used.progress - for reporting progress if it is not null.IOException - indicating that append is not supported.public boolean rename(org.apache.hadoop.fs.Path src,
org.apache.hadoop.fs.Path dst)
throws IOException
Fails if src is a file and dst is a directory.
Fails if src is a directory and dst is a file.
Fails if the parent of dst does not exist or is a file.
Fails if dst is a directory that is not empty.
rename in class org.apache.hadoop.fs.FileSystemsrc - path to be renameddst - new path after renameIOException - on IO failurepublic com.amazonaws.services.s3.model.ObjectMetadata getObjectMetadata(org.apache.hadoop.fs.Path path)
throws IOException
path - path to the objectIOException - IO and object access problems.public boolean hasMetadataStore()
public MetadataStore getMetadataStore()
NullMetadataStore.protected void incrementStatistic(Statistic statistic)
statistic - The operation to incrementprotected void incrementStatistic(Statistic statistic, long count)
statistic - The operation to incrementcount - the count to incrementprotected void decrementGauge(Statistic statistic, long count)
statistic - The operation to decrementcount - the count to decrementprotected void incrementGauge(Statistic statistic, long count)
statistic - The operation to incrementcount - the count to incrementpublic S3AStorageStatistics getStorageStatistics()
getStorageStatistics in class org.apache.hadoop.fs.FileSystemprotected com.amazonaws.services.s3.model.ObjectMetadata getObjectMetadata(String key)
key - keyprotected S3ListResult listObjects(S3ListRequest request)
listObjects operation, incrementing metrics
in the process.request - request to initiateprotected S3ListResult continueListObjects(S3ListRequest request, S3ListResult prevResult)
request - last list objects request to continueprevResult - last paged result to continue frompublic void incrementReadOperations()
public void incrementWriteOperations()
public com.amazonaws.services.s3.model.PutObjectRequest newPutObjectRequest(String key, com.amazonaws.services.s3.model.ObjectMetadata metadata, File srcfile)
key - key of objectmetadata - metadata headersrcfile - source filepublic com.amazonaws.services.s3.model.ObjectMetadata newObjectMetadata()
public com.amazonaws.services.s3.model.ObjectMetadata newObjectMetadata(long length)
length - length of data to set in header.public UploadInfo putObject(com.amazonaws.services.s3.model.PutObjectRequest putObjectRequest)
putObjectRequest - the requestpublic com.amazonaws.services.s3.model.UploadPartResult uploadPart(com.amazonaws.services.s3.model.UploadPartRequest request)
throws com.amazonaws.AmazonClientException
request - requestcom.amazonaws.AmazonClientException - on problemspublic void incrementPutStartStatistics(long bytes)
bytes - bytes in the request.public void incrementPutCompletedStatistics(boolean success,
long bytes)
success - did the operation succeed?bytes - bytes in the request.public void incrementPutProgressStatistics(String key, long bytes)
key - key to file that is being written (for logging)bytes - bytes successfully uploaded.public boolean delete(org.apache.hadoop.fs.Path f,
boolean recursive)
throws IOException
O(files), with
added overheads to enumerate the path. It is also not atomic.delete in class org.apache.hadoop.fs.FileSystemf - the path to delete.recursive - if path is a directory and set to
true, the directory is deleted else throws an exception. In
case of a file the recursive can be set to either true or false.IOException - due to inability to delete a directory or file.public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path f)
throws FileNotFoundException,
IOException
listStatus in class org.apache.hadoop.fs.FileSystemf - given pathFileNotFoundException - when the path does not exist;
IOException see specific implementationIOExceptionpublic org.apache.hadoop.fs.FileStatus[] innerListStatus(org.apache.hadoop.fs.Path f)
throws FileNotFoundException,
IOException,
com.amazonaws.AmazonClientException
f - given pathFileNotFoundException - when the path does not exist;IOException - due to an IO problem.com.amazonaws.AmazonClientException - on failures inside the AWS SDKpublic void setWorkingDirectory(org.apache.hadoop.fs.Path newDir)
setWorkingDirectory in class org.apache.hadoop.fs.FileSystemnewDir - the current working directory.public org.apache.hadoop.fs.Path getWorkingDirectory()
getWorkingDirectory in class org.apache.hadoop.fs.FileSystempublic String getUsername()
public boolean mkdirs(org.apache.hadoop.fs.Path path,
org.apache.hadoop.fs.permission.FsPermission permission)
throws IOException,
org.apache.hadoop.fs.FileAlreadyExistsException
'mkdir -p'.
Existence of the directory hierarchy is not an error.mkdirs in class org.apache.hadoop.fs.FileSystempath - path to createpermission - to apply to forg.apache.hadoop.fs.FileAlreadyExistsException - there is a file at the path specifiedIOException - other IO problemspublic org.apache.hadoop.fs.FileStatus getFileStatus(org.apache.hadoop.fs.Path f)
throws IOException
getFileStatus in class org.apache.hadoop.fs.FileSystemf - The path we want information fromFileNotFoundException - when the path does not existIOException - on other problems.public void copyFromLocalFile(boolean delSrc,
boolean overwrite,
org.apache.hadoop.fs.Path src,
org.apache.hadoop.fs.Path dst)
throws IOException
copyFromLocalFile in class org.apache.hadoop.fs.FileSystemdelSrc - whether to delete the srcoverwrite - whether to overwrite an existing filesrc - pathdst - pathIOException - IO problemorg.apache.hadoop.fs.FileAlreadyExistsException - the destination file exists and
overwrite==falsecom.amazonaws.AmazonClientException - failure in the AWS SDKpublic void close()
throws IOException
close in interface Closeableclose in interface AutoCloseableclose in class org.apache.hadoop.fs.FileSystemIOException - IO problempublic String getCanonicalServiceName()
getCanonicalServiceName in class org.apache.hadoop.fs.FileSystemprotected void setOptionalMultipartUploadRequestParameters(com.amazonaws.services.s3.model.InitiateMultipartUploadRequest req)
protected void setOptionalUploadPartRequestParameters(com.amazonaws.services.s3.model.UploadPartRequest request)
request - upload part requestprotected void setOptionalCopyObjectRequestParameters(com.amazonaws.services.s3.model.CopyObjectRequest copyObjectRequest)
throws IOException
IOException@Deprecated public long getDefaultBlockSize()
FileSystem.getDefaultBlockSize(Path) insteadgetDefaultBlockSize in class org.apache.hadoop.fs.FileSystempublic long getPartitionSize()
public long getMultiPartThreshold()
public org.apache.hadoop.fs.FileStatus[] globStatus(org.apache.hadoop.fs.Path pathPattern)
throws IOException
Statistic.INVOCATION_GLOB_STATUS.
globStatus in class org.apache.hadoop.fs.FileSystemIOExceptionpublic org.apache.hadoop.fs.FileStatus[] globStatus(org.apache.hadoop.fs.Path pathPattern,
org.apache.hadoop.fs.PathFilter filter)
throws IOException
globStatus in class org.apache.hadoop.fs.FileSystemIOExceptionpublic boolean exists(org.apache.hadoop.fs.Path f)
throws IOException
exists in class org.apache.hadoop.fs.FileSystemIOExceptionpublic boolean isDirectory(org.apache.hadoop.fs.Path f)
throws IOException
isDirectory in class org.apache.hadoop.fs.FileSystemIOExceptionpublic boolean isFile(org.apache.hadoop.fs.Path f)
throws IOException
isFile in class org.apache.hadoop.fs.FileSystemIOExceptionpublic org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listFiles(org.apache.hadoop.fs.Path f,
boolean recursive)
throws FileNotFoundException,
IOException
ListObjectsRequest is created requesting a (windowed)
listing of all entries under the given path. This is used to construct
an ObjectListingIterator instance, iteratively returning the
sequence of lists of elements under the path. This is then iterated
over in a FileStatusListingIterator, which generates
S3AFileStatus instances, one per listing entry.
These are then translated into LocatedFileStatus instances.
This is essentially a nested and wrapped set of iterators, with some
generator classes; an architecture which may become less convoluted
using lambda-expressions.listFiles in class org.apache.hadoop.fs.FileSystemf - a pathrecursive - if the subdirectories need to be traversed recursivelyFileNotFoundException - if path does not existIOException - if any I/O error occurredpublic org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listFilesAndEmptyDirectories(org.apache.hadoop.fs.Path f,
boolean recursive)
throws IOException
IOExceptionpublic org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listLocatedStatus(org.apache.hadoop.fs.Path f)
throws FileNotFoundException,
IOException
listLocatedStatus in class org.apache.hadoop.fs.FileSystemFileNotFoundExceptionIOExceptionpublic org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listLocatedStatus(org.apache.hadoop.fs.Path f,
org.apache.hadoop.fs.PathFilter filter)
throws FileNotFoundException,
IOException
listLocatedStatus in class org.apache.hadoop.fs.FileSystemf - a pathfilter - a path filterFileNotFoundException - if path does not existIOException - if any I/O error occurredCopyright © 2008–2018 Apache Software Foundation. All rights reserved.