@InterfaceAudience.Private @InterfaceStability.Unstable public final class S3Guard extends Object
| Modifier and Type | Field and Description |
|---|---|
static String |
S3GUARD_DDB_CLIENT_FACTORY_IMPL |
| Modifier and Type | Method and Description |
|---|---|
static void |
addAncestors(MetadataStore metadataStore,
org.apache.hadoop.fs.Path qualifiedPath,
String username) |
static void |
addMoveAncestors(MetadataStore ms,
Collection<org.apache.hadoop.fs.Path> srcPaths,
Collection<PathMetadata> dstMetas,
org.apache.hadoop.fs.Path srcRoot,
org.apache.hadoop.fs.Path srcPath,
org.apache.hadoop.fs.Path dstPath,
String owner)
Helper method that records the move of all ancestors of a path.
|
static void |
addMoveDir(MetadataStore ms,
Collection<org.apache.hadoop.fs.Path> srcPaths,
Collection<PathMetadata> dstMetas,
org.apache.hadoop.fs.Path srcPath,
org.apache.hadoop.fs.Path dstPath,
String owner)
Helper function that records the move of directory paths, adding
resulting metadata to the supplied lists.
|
static void |
addMoveFile(MetadataStore ms,
Collection<org.apache.hadoop.fs.Path> srcPaths,
Collection<PathMetadata> dstMetas,
org.apache.hadoop.fs.Path srcPath,
org.apache.hadoop.fs.Path dstPath,
long size,
long blockSize,
String owner)
Like
addMoveDir(MetadataStore, Collection, Collection, Path,
Path, String) (), but for files. |
static void |
assertQualified(org.apache.hadoop.fs.Path... paths)
Assert that all paths are valid.
|
static void |
assertQualified(org.apache.hadoop.fs.Path p)
Assert that the path is qualified with a host and scheme.
|
static org.apache.hadoop.fs.FileStatus[] |
dirListingUnion(MetadataStore ms,
org.apache.hadoop.fs.Path path,
List<org.apache.hadoop.fs.FileStatus> backingStatuses,
DirListingMetadata dirMeta,
boolean isAuthoritative)
Given directory listing metadata from both the backing store and the
MetadataStore, merge the two sources of truth to create a consistent
view of the current directory contents, which can be returned to clients.
|
static org.apache.hadoop.fs.FileStatus[] |
dirMetaToStatuses(DirListingMetadata dirMeta)
Convert the data of a directory listing to an array of
FileStatus
entries. |
static MetadataStore |
getMetadataStore(org.apache.hadoop.fs.FileSystem fs)
Create a new instance of the configured MetadataStore.
|
static boolean |
isNullMetadataStore(MetadataStore ms)
Although NullMetadataStore does nothing, callers may wish to avoid work
(fast path) when the NullMetadataStore is in use.
|
static void |
makeDirsOrdered(MetadataStore ms,
List<org.apache.hadoop.fs.Path> dirs,
String owner,
boolean authoritative)
Update MetadataStore to reflect creation of the given directories.
|
static S3AFileStatus |
putAndReturn(MetadataStore ms,
S3AFileStatus status,
S3AInstrumentation instrumentation)
Helper function which puts a given S3AFileStatus into the MetadataStore and
returns the same S3AFileStatus.
|
@InterfaceAudience.Private @InterfaceStability.Unstable public static final String S3GUARD_DDB_CLIENT_FACTORY_IMPL
public static MetadataStore getMetadataStore(org.apache.hadoop.fs.FileSystem fs) throws IOException
MetadataStore.initialize(FileSystem) by this function before
returning it. Callers must clean up by calling
Closeable.close() when done using the MetadataStore.fs - FileSystem whose Configuration specifies which
implementation to use.IOException - if the metadata store cannot be instantiatedpublic static S3AFileStatus putAndReturn(MetadataStore ms, S3AFileStatus status, S3AInstrumentation instrumentation) throws IOException
ms - MetadataStore to put() into.status - status to storeinstrumentation - instrumentation of the s3a file systemIOException - if metadata store update failedpublic static org.apache.hadoop.fs.FileStatus[] dirMetaToStatuses(DirListingMetadata dirMeta)
FileStatus
entries. Tombstones are filtered out at this point. If the listing is null
an empty array is returned.dirMeta - directory listing -may be nullpublic static org.apache.hadoop.fs.FileStatus[] dirListingUnion(MetadataStore ms, org.apache.hadoop.fs.Path path, List<org.apache.hadoop.fs.FileStatus> backingStatuses, DirListingMetadata dirMeta, boolean isAuthoritative) throws IOException
ms - MetadataStore to use.path - path to directorybackingStatuses - Directory listing from the backing store.dirMeta - Directory listing from MetadataStore. May be null.isAuthoritative - State of authoritative modeIOException - if metadata store update failedpublic static boolean isNullMetadataStore(MetadataStore ms)
ms - The MetadataStore to testpublic static void makeDirsOrdered(MetadataStore ms, List<org.apache.hadoop.fs.Path> dirs, String owner, boolean authoritative)
ms - MetadataStore to update.dirs - null, or an ordered list of directories from leaf to root.
E.g. if /a/ exists, and mkdirs(/a/b/c/d) is called, this
list will contain [/a/b/c/d, /a/b/c, /a/b]. /a/b/c/d is
an empty, dir, and the other dirs only contain their child
dir.owner - Hadoop user name.authoritative - Whether to mark new directories as authoritative.public static void addMoveDir(MetadataStore ms, Collection<org.apache.hadoop.fs.Path> srcPaths, Collection<PathMetadata> dstMetas, org.apache.hadoop.fs.Path srcPath, org.apache.hadoop.fs.Path dstPath, String owner)
ms - MetadataStore, used to make this a no-op, when it is
NullMetadataStore.srcPaths - stores the source path heredstMetas - stores destination metadata heresrcPath - source path to storedstPath - destination path to storeowner - file owner to use in created recordspublic static void addMoveFile(MetadataStore ms, Collection<org.apache.hadoop.fs.Path> srcPaths, Collection<PathMetadata> dstMetas, org.apache.hadoop.fs.Path srcPath, org.apache.hadoop.fs.Path dstPath, long size, long blockSize, String owner)
addMoveDir(MetadataStore, Collection, Collection, Path,
Path, String) (), but for files.ms - MetadataStore, used to make this a no-op, when it is
NullMetadataStore.srcPaths - stores the source path heredstMetas - stores destination metadata heresrcPath - source path to storedstPath - destination path to storesize - length of file movedblockSize - blocksize to associate with destination fileowner - file owner to use in created recordspublic static void addMoveAncestors(MetadataStore ms, Collection<org.apache.hadoop.fs.Path> srcPaths, Collection<PathMetadata> dstMetas, org.apache.hadoop.fs.Path srcRoot, org.apache.hadoop.fs.Path srcPath, org.apache.hadoop.fs.Path dstPath, String owner)
addMoveFile(org.apache.hadoop.fs.s3a.s3guard.MetadataStore, java.util.Collection<org.apache.hadoop.fs.Path>, java.util.Collection<org.apache.hadoop.fs.s3a.s3guard.PathMetadata>, org.apache.hadoop.fs.Path, org.apache.hadoop.fs.Path, long, long, java.lang.String) and addMoveDir(org.apache.hadoop.fs.s3a.s3guard.MetadataStore, java.util.Collection<org.apache.hadoop.fs.Path>, java.util.Collection<org.apache.hadoop.fs.s3a.s3guard.PathMetadata>, org.apache.hadoop.fs.Path, org.apache.hadoop.fs.Path, java.lang.String), this method adds resulting
metadata to the supplied lists. It does not store in MetadataStore.ms - MetadataStore, no-op if it is NullMetadataStoresrcPaths - stores the source path heredstMetas - stores destination metadata heresrcRoot - source root up to which (exclusive) should we add ancestorssrcPath - source path of the child to add ancestorsdstPath - destination path of the child to add ancestorsowner - Hadoop user namepublic static void addAncestors(MetadataStore metadataStore, org.apache.hadoop.fs.Path qualifiedPath, String username) throws IOException
IOExceptionpublic static void assertQualified(org.apache.hadoop.fs.Path p)
p - path to checkNullPointerException - if either argument does not holdpublic static void assertQualified(org.apache.hadoop.fs.Path... paths)
paths - path to checkNullPointerException - if either argument does not holdCopyright © 2008–2018 Apache Software Foundation. All rights reserved.