| Package | Description |
|---|---|
| org.apache.hadoop.fs.s3a |
S3A Filesystem.
|
| org.apache.hadoop.fs.s3a.s3guard |
This package contains classes related to S3Guard: a feature of S3A to mask
the eventual consistency behavior of S3 and optimize access patterns by
coordinating with a strongly consistent external store for file system
metadata.
|
| Modifier and Type | Method and Description |
|---|---|
MetadataStore |
S3AFileSystem.getMetadataStore()
Get the metadata store.
|
| Modifier and Type | Class and Description |
|---|---|
class |
DynamoDBMetadataStore
DynamoDBMetadataStore is a
MetadataStore that persists
file system metadata to DynamoDB. |
class |
LocalMetadataStore
This is a local, in-memory, implementation of MetadataStore.
|
class |
NullMetadataStore
A no-op implementation of MetadataStore.
|
| Modifier and Type | Method and Description |
|---|---|
static MetadataStore |
S3Guard.getMetadataStore(org.apache.hadoop.fs.FileSystem fs)
Create a new instance of the configured MetadataStore.
|
MetadataStore |
S3GuardTool.getStore() |
| Modifier and Type | Method and Description |
|---|---|
static void |
S3Guard.addAncestors(MetadataStore metadataStore,
org.apache.hadoop.fs.Path qualifiedPath,
String username) |
static void |
S3Guard.addMoveAncestors(MetadataStore ms,
Collection<org.apache.hadoop.fs.Path> srcPaths,
Collection<PathMetadata> dstMetas,
org.apache.hadoop.fs.Path srcRoot,
org.apache.hadoop.fs.Path srcPath,
org.apache.hadoop.fs.Path dstPath,
String owner)
Helper method that records the move of all ancestors of a path.
|
static void |
S3Guard.addMoveDir(MetadataStore ms,
Collection<org.apache.hadoop.fs.Path> srcPaths,
Collection<PathMetadata> dstMetas,
org.apache.hadoop.fs.Path srcPath,
org.apache.hadoop.fs.Path dstPath,
String owner)
Helper function that records the move of directory paths, adding
resulting metadata to the supplied lists.
|
static void |
S3Guard.addMoveFile(MetadataStore ms,
Collection<org.apache.hadoop.fs.Path> srcPaths,
Collection<PathMetadata> dstMetas,
org.apache.hadoop.fs.Path srcPath,
org.apache.hadoop.fs.Path dstPath,
long size,
long blockSize,
String owner)
Like
S3Guard.addMoveDir(MetadataStore, Collection, Collection, Path,
Path, String) (), but for files. |
static org.apache.hadoop.fs.FileStatus[] |
S3Guard.dirListingUnion(MetadataStore ms,
org.apache.hadoop.fs.Path path,
List<org.apache.hadoop.fs.FileStatus> backingStatuses,
DirListingMetadata dirMeta,
boolean isAuthoritative)
Given directory listing metadata from both the backing store and the
MetadataStore, merge the two sources of truth to create a consistent
view of the current directory contents, which can be returned to clients.
|
static boolean |
S3Guard.isNullMetadataStore(MetadataStore ms)
Although NullMetadataStore does nothing, callers may wish to avoid work
(fast path) when the NullMetadataStore is in use.
|
static void |
S3Guard.makeDirsOrdered(MetadataStore ms,
List<org.apache.hadoop.fs.Path> dirs,
String owner,
boolean authoritative)
Update MetadataStore to reflect creation of the given directories.
|
protected static void |
S3GuardTool.printStoreDiagnostics(PrintStream out,
MetadataStore store)
Retrieve and Print store diagnostics.
|
static S3AFileStatus |
S3Guard.putAndReturn(MetadataStore ms,
S3AFileStatus status,
S3AInstrumentation instrumentation)
Helper function which puts a given S3AFileStatus into the MetadataStore and
returns the same S3AFileStatus.
|
protected void |
S3GuardTool.setStore(MetadataStore store) |
| Constructor and Description |
|---|
DescendantsIterator(MetadataStore ms,
PathMetadata meta)
Creates a new
DescendantsIterator. |
MetadataStoreListFilesIterator(MetadataStore ms,
PathMetadata meta,
boolean allowAuthoritative) |
Copyright © 2008–2018 Apache Software Foundation. All rights reserved.