com.google.cloud.hadoop.gcsio

Class GoogleCloudStorageFileSystem

java.lang.Object

com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem

public class GoogleCloudStorageFileSystem
extends java.lang.Object

Provides a POSIX like file system layered on top of Google Cloud Storage (GCS). All file system aspects (eg, path) are encapsulated in this class, they are not exposed to the underlying layer. That is, all interactions with the underlying layer are strictly in terms of buckets and objects.

Field Summary

Fields

Modifier and Type	Field and Description
static java.net.URI	GCS_ROOT
static PathCodec	LEGACY_PATH_CODEC A PathCodec that maintains compatibility with versions of GCS FS < 1.4.5.
static org.slf4j.Logger	LOG
static java.lang.String	SCHEME
static PathCodec	URI_ENCODED_PATH_CODEC A PathCodec that expects URIs to be of the form: gs://authority/properly/encoded/path.

Constructor Summary

Constructors

Constructor and Description
GoogleCloudStorageFileSystem(com.google.api.client.auth.oauth2.Credential credential, GoogleCloudStorageFileSystemOptions options) Constructs an instance of GoogleCloudStorageFileSystem.
GoogleCloudStorageFileSystem(GoogleCloudStorage gcs) Constructs a GoogleCloudStorageFilesystem based on an already-configured underlying GoogleCloudStorage gcs.
GoogleCloudStorageFileSystem(GoogleCloudStorage gcs, GoogleCloudStorageFileSystemOptions options) Constructs a GoogleCloudStorageFilesystem based on an already-configured underlying GoogleCloudStorage gcs.

Method Summary

Modifier and Type	Method and Description
void	close() Releases resources used by this instance.
void	compose(java.util.List<java.net.URI> sources, java.net.URI destination, java.lang.String contentType) Composes inputs into a single GCS object.
java.nio.channels.WritableByteChannel	create(java.net.URI path) Creates and opens an object for writing.
java.nio.channels.WritableByteChannel	create(java.net.URI path, CreateFileOptions options) Creates and opens an object for writing.
void	delete(java.net.URI path, boolean recursive) Deletes one or more items indicated by the given path.
boolean	exists(java.net.URI path) Indicates whether the given item exists.
FileInfo	getFileInfo(java.net.URI path) Gets information about the given path item.
java.util.List<FileInfo>	getFileInfos(java.util.List<java.net.URI> paths) Gets information about each path in the given list; more efficient than calling getFileInfo() on each path individually in a loop.
GoogleCloudStorage	getGcs() Retrieve our internal gcs.
GoogleCloudStorageFileSystemOptions	getOptions() Retrieve the options that were used to create this GoogleCloudStorageFileSystem.
static java.net.URI	getParentPath(PathCodec pathCodec, java.net.URI path) Deprecated. This static method is included as a transitional utility and the instance method variant should be preferred.
java.net.URI	getParentPath(java.net.URI path) Gets the parent directory of the given path.
static java.net.URI	getPath(java.lang.String bucketName) Deprecated. This method is deprecated as each instance of GCS FS can be configured with a codec.
static java.net.URI	getPath(java.lang.String bucketName, java.lang.String objectName) Deprecated. This method is deprecated as each instance of GCS FS can be configured with a codec.
static java.net.URI	getPath(java.lang.String bucketName, java.lang.String objectName, boolean allowEmptyObjectName) Deprecated. This method is deprecated as each instance of GCS FS can be configured with a codec.
PathCodec	getPathCodec() The PathCodec in use by this file system.
java.util.List<FileInfo>	listAllFileInfoForPrefix(java.net.URI prefix) Equivalent to a recursive listing of prefix, except that prefix doesn't have to represent an actual object but can just be a partial prefix string, and there is no auto-repair of implicit directories since we can't detect implicit directories without listing by 'delimiter'.
java.util.List<FileInfo>	listFileInfo(java.net.URI path, boolean enableAutoRepair) If the given path points to a directory then the information about its children is returned, otherwise information about the given file is returned.
java.util.List<java.net.URI>	listFileNames(FileInfo fileInfo) If the given item is a directory then the paths of its immediate children are returned, otherwise the path of the given item is returned.
java.util.List<java.net.URI>	listFileNames(FileInfo fileInfo, boolean recursive) If the given item is a directory then the paths of its children are returned, otherwise the path of the given item is returned.
void	mkdir(java.net.URI path) Creates a directory at the specified path.
void	mkdirs(java.net.URI path) Creates a directory at the specified path.
static CreateObjectOptions	objectOptionsFromFileOptions(CreateFileOptions options) Convert CreateFileOptions to CreateObjectOptions.
java.nio.channels.SeekableByteChannel	open(java.net.URI path) Opens an object for reading.
java.nio.channels.SeekableByteChannel	open(java.net.URI path, GoogleCloudStorageReadOptions readOptions) Opens an object for reading.
void	rename(java.net.URI src, java.net.URI dst) Renames the given item's path.
void	repairDirs(java.util.List<java.net.URI> exactDirPaths) Creates the list of directories specified in exactDirPaths; doesn't perform validation and doesn't create their parent dirs if their parent dirs don't already exist.
boolean	repairPossibleImplicitDirectory(java.net.URI path) Checks that path doesn't already exist as a directory object, and if so, performs an object listing using the full path as the match prefix so that if there are any objects that imply path is a parent directory, we will discover its existence as a returned GCS 'prefix'.
protected void	tryUpdateTimestampsForParentDirectories(java.util.List<java.net.URI> modifiedObjects, java.util.List<java.net.URI> excludedParents) For each listed modified object, attempt to update the modification time of the parent directory.
protected void	updateTimestampsForParentDirectories(java.util.List<java.net.URI> modifiedObjects, java.util.List<java.net.URI> excludedParents) For each listed modified object, attempt to update the modification time of the parent directory.
static StorageResourceId	validatePathAndGetId(java.net.URI uri, boolean allowEmptyObjectNames) Deprecated. This method is deprecated as each instance of GCS FS can be configured with a codec.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SCHEME

public static final java.lang.String SCHEME

See Also:

Constant Field Values

GCS_ROOT

public static final java.net.URI GCS_ROOT

LOG

public static final org.slf4j.Logger LOG

LEGACY_PATH_CODEC

public static final PathCodec LEGACY_PATH_CODEC

A PathCodec that maintains compatibility with versions of GCS FS < 1.4.5.

URI_ENCODED_PATH_CODEC

public static final PathCodec URI_ENCODED_PATH_CODEC

A PathCodec that expects URIs to be of the form: gs://authority/properly/encoded/path.

Constructor Detail

GoogleCloudStorageFileSystem

public GoogleCloudStorageFileSystem(com.google.api.client.auth.oauth2.Credential credential,
                                    GoogleCloudStorageFileSystemOptions options)
                             throws java.io.IOException

Constructs an instance of GoogleCloudStorageFileSystem.

Parameters:

credential - OAuth2 credential that allows access to GCS.

options - Options for how this filesystem should operate and configure its underlying storage.

Throws:

java.io.IOException

GoogleCloudStorageFileSystem

public GoogleCloudStorageFileSystem(GoogleCloudStorage gcs)
                             throws java.io.IOException

Constructs a GoogleCloudStorageFilesystem based on an already-configured underlying GoogleCloudStorage gcs.

Throws:

java.io.IOException

GoogleCloudStorageFileSystem

public GoogleCloudStorageFileSystem(GoogleCloudStorage gcs,
                                    GoogleCloudStorageFileSystemOptions options)
                             throws java.io.IOException

Constructs a GoogleCloudStorageFilesystem based on an already-configured underlying GoogleCloudStorage gcs. Any options pertaining to GCS creation will be ignored.

Throws:

java.io.IOException

Method Detail

getOptions

public GoogleCloudStorageFileSystemOptions getOptions()

Retrieve the options that were used to create this GoogleCloudStorageFileSystem.

create

public java.nio.channels.WritableByteChannel create(java.net.URI path)
                                             throws java.io.IOException

Creates and opens an object for writing. If the object already exists, it is deleted.

Parameters:

path - Object full path of the form gs://bucket/object-path.

Returns:

A channel for writing to the given object.

Throws:

java.io.IOException

objectOptionsFromFileOptions

public static CreateObjectOptions objectOptionsFromFileOptions(CreateFileOptions options)

Convert CreateFileOptions to CreateObjectOptions.

create

public java.nio.channels.WritableByteChannel create(java.net.URI path,
                                                    CreateFileOptions options)
                                             throws java.io.IOException

Creates and opens an object for writing.

Parameters:

path - Object full path of the form gs://bucket/object-path.

Returns:

A channel for writing to the given object.

Throws:

java.io.IOException

open

public java.nio.channels.SeekableByteChannel open(java.net.URI path)
                                           throws java.io.IOException

Opens an object for reading.

Parameters:

path - Object full path of the form gs://bucket/object-path.

Returns:

A channel for reading from the given object.

Throws:

java.io.FileNotFoundException - if the given path does not exist.

java.io.IOException - if object exists but cannot be opened.

open

public java.nio.channels.SeekableByteChannel open(java.net.URI path,
                                                  GoogleCloudStorageReadOptions readOptions)
                                           throws java.io.IOException

Opens an object for reading.

Parameters:

path - Object full path of the form gs://bucket/object-path.

readOptions - Fine-grained options for behaviors of retries, buffering, etc.

Returns:

A channel for reading from the given object.

Throws:

java.io.FileNotFoundException - if the given path does not exist.

java.io.IOException - if object exists but cannot be opened.

delete

public void delete(java.net.URI path,
                   boolean recursive)
            throws java.io.IOException

Deletes one or more items indicated by the given path. If path points to a directory: -- if recursive is true, all items under that path are recursively deleted followed by deletion of the directory. -- else, -- the directory is deleted if it is empty, -- else, an IOException is thrown. The recursive parameter is ignored for a file.

Parameters:

path - Path of the item to delete.

recursive - If true, all sub-items are also deleted.

Throws:

java.io.FileNotFoundException - if the given path does not exist.

java.io.IOException

exists

public boolean exists(java.net.URI path)
               throws java.io.IOException

Indicates whether the given item exists.

Parameters:

path - Path of the item to check.

Returns:

true if the given item exists, false otherwise.

Throws:

java.io.IOException

repairDirs

public void repairDirs(java.util.List<java.net.URI> exactDirPaths)
                throws java.io.IOException

Creates the list of directories specified in exactDirPaths; doesn't perform validation and doesn't create their parent dirs if their parent dirs don't already exist. Use with caution.

Throws:

java.io.IOException

mkdirs

public void mkdirs(java.net.URI path)
            throws java.io.IOException

Creates a directory at the specified path. Also creates any parent directories as necessary. Similar to 'mkdir -p' command.

Parameters:

path - Path of the directory to create.

Throws:

java.io.IOException

rename

public void rename(java.net.URI src,
                   java.net.URI dst)
            throws java.io.IOException

Renames the given item's path. The operation is disallowed if any of the following is true: -- src == GCS_ROOT -- src is a file and dst == GCS_ROOT -- src does not exist -- dst is a file that already exists -- parent of the destination does not exist. Otherwise, the expected behavior is as follows: -- if src is a directory -- dst is an existing file => disallowed -- dst is a directory => rename the directory. -- if src is a file -- dst is a file => rename the file. -- dst is a directory => similar to the previous case after appending src file-name to dst Note: This function is very expensive to call for directories that have many sub-items.

Parameters:

src - Path of the item to rename.

dst - New path of the item.

Throws:

java.io.FileNotFoundException - if src does not exist.

java.io.IOException

compose

public void compose(java.util.List<java.net.URI> sources,
                    java.net.URI destination,
                    java.lang.String contentType)
             throws java.io.IOException

Composes inputs into a single GCS object. This performs a GCS Compose. Objects will be composed according to the order they appear in the input. The destination object, if already present, will be overwritten. Sources and destination are assumed to be in the same bucket.

Parameters:

sources - the list of URIs to be composed

destination - the resulting URI with composed sources

contentType - content-type of the composed object

Throws:

java.io.IOException - if the Compose operation was unsuccessful

listFileNames

public java.util.List<java.net.URI> listFileNames(FileInfo fileInfo)
                                           throws java.io.IOException

If the given item is a directory then the paths of its immediate children are returned, otherwise the path of the given item is returned.

Parameters:

fileInfo - FileInfo of an item.

Returns:

Paths of children (if directory) or self path.

Throws:

java.io.IOException

listFileNames

public java.util.List<java.net.URI> listFileNames(FileInfo fileInfo,
                                                  boolean recursive)
                                           throws java.io.IOException

If the given item is a directory then the paths of its children are returned, otherwise the path of the given item is returned.

Parameters:

fileInfo - FileInfo of an item.

recursive - If true, path of all children are returned; else, only immediate children are returned.

Returns:

Paths of children (if directory) or self path.

Throws:

java.io.IOException

repairPossibleImplicitDirectory

public boolean repairPossibleImplicitDirectory(java.net.URI path)
                                        throws java.io.IOException

Checks that path doesn't already exist as a directory object, and if so, performs an object listing using the full path as the match prefix so that if there are any objects that imply path is a parent directory, we will discover its existence as a returned GCS 'prefix'. In such a case, the directory object will be explicitly created.

Returns:

true if a repair was successfully made, false if a repair was unnecessary or failed.

Throws:

java.io.IOException

listAllFileInfoForPrefix

public java.util.List<FileInfo> listAllFileInfoForPrefix(java.net.URI prefix)
                                                  throws java.io.IOException

Equivalent to a recursive listing of prefix, except that prefix doesn't have to represent an actual object but can just be a partial prefix string, and there is no auto-repair of implicit directories since we can't detect implicit directories without listing by 'delimiter'. The 'authority' component of the prefix *must* be the complete authority, however; we can only list prefixes of *objects*, not buckets.

Parameters:

prefix - the prefix to use to list all matching objects.

Throws:

java.io.IOException

listFileInfo

public java.util.List<FileInfo> listFileInfo(java.net.URI path,
                                             boolean enableAutoRepair)
                                      throws java.io.IOException

If the given path points to a directory then the information about its children is returned, otherwise information about the given file is returned. Note: This function is expensive to call, especially for a directory with many children. Use the alternative listFileNames(FileInfo) if you only need names of children and no other attributes.

Parameters:

path - Given path.

enableAutoRepair - if true, attempt to repair implicit directories when detected.

Returns:

Information about a file or children of a directory.

Throws:

java.io.FileNotFoundException - if the given path does not exist.

java.io.IOException

getFileInfo

public FileInfo getFileInfo(java.net.URI path)
                     throws java.io.IOException

Gets information about the given path item.

Parameters:

path - The path we want information about.

Returns:

Information about the given path item.

Throws:

java.io.IOException

getFileInfos

public java.util.List<FileInfo> getFileInfos(java.util.List<java.net.URI> paths)
                                      throws java.io.IOException

Gets information about each path in the given list; more efficient than calling getFileInfo() on each path individually in a loop.

Parameters:

paths - List of paths.

Returns:

Information about each path in the given list.

Throws:

java.io.IOException

close

public void close()

Releases resources used by this instance.

mkdir

public void mkdir(java.net.URI path)
           throws java.io.IOException

Creates a directory at the specified path. There are two conventions for using objects as directories in GCS. 1. An object of zero size with name ending in / 2. An object of zero size with name ending in _$folder$ #1 is the recommended convention by the GCS team. We use it when creating a directory. However, some old tools still use #2. We will decide based on customer use cases whether to support #2 as well. For now, we only support #1. Note that a bucket is always considered a directory. Doesn't create parent directories; normal use cases should only call mkdirs().

Throws:

java.io.IOException

updateTimestampsForParentDirectories

protected void updateTimestampsForParentDirectories(java.util.List<java.net.URI> modifiedObjects,
                                                    java.util.List<java.net.URI> excludedParents)
                                             throws java.io.IOException

For each listed modified object, attempt to update the modification time of the parent directory.

Parameters:

modifiedObjects - The objects that have been modified

excludedParents - A list of parent directories that we shouldn't attempt to update.

Throws:

java.io.IOException

tryUpdateTimestampsForParentDirectories

protected void tryUpdateTimestampsForParentDirectories(java.util.List<java.net.URI> modifiedObjects,
                                                       java.util.List<java.net.URI> excludedParents)

For each listed modified object, attempt to update the modification time of the parent directory. This method will log & swallow exceptions thrown by the GCSIO layer.

Parameters:

modifiedObjects - The objects that have been modified

excludedParents - A list of parent directories that we shouldn't attempt to update.

getParentPath

public java.net.URI getParentPath(java.net.URI path)

Gets the parent directory of the given path.

Parameters:

path - Path to convert.

Returns:

Path of parent directory of the given item or null for root path.

getGcs

public GoogleCloudStorage getGcs()

Retrieve our internal gcs.

getPathCodec

public PathCodec getPathCodec()

The PathCodec in use by this file system.

validatePathAndGetId

@Deprecated
public static StorageResourceId validatePathAndGetId(java.net.URI uri,
                                                                 boolean allowEmptyObjectNames)

Deprecated. This method is deprecated as each instance of GCS FS can be configured with a codec.

Validate a URI using the legacy path codec and return a StorageResourceId.

getPath

@Deprecated
public static java.net.URI getPath(java.lang.String bucketName,
                                               java.lang.String objectName,
                                               boolean allowEmptyObjectName)

Deprecated. This method is deprecated as each instance of GCS FS can be configured with a codec.

Construct a URI using the legacy path codec.

getPath

@Deprecated
public static java.net.URI getPath(java.lang.String bucketName)

Deprecated. This method is deprecated as each instance of GCS FS can be configured with a codec.

Construct a URI using the legacy path codec.

getPath

@Deprecated
public static java.net.URI getPath(java.lang.String bucketName,
                                               java.lang.String objectName)

Deprecated. This method is deprecated as each instance of GCS FS can be configured with a codec.

Construct a URI using the legacy path codec.

getParentPath

@Deprecated
public static java.net.URI getParentPath(PathCodec pathCodec,
                                                     java.net.URI path)

Deprecated. This static method is included as a transitional utility and the instance method variant should be preferred.

Gets the parent directory of the given path.

Parameters:

path - Path to convert.

Returns:

Path of parent directory of the given item or null for root path.