Package com.google.cloud.hadoop.fs.gcs

Class GoogleHadoopFileSystem

java.lang.Object
org.apache.hadoop.conf.Configured
org.apache.hadoop.fs.FileSystem
com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem
All Implemented Interfaces:
Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable, org.apache.hadoop.fs.BulkDeleteSource, org.apache.hadoop.fs.PathCapabilities, org.apache.hadoop.fs.statistics.IOStatisticsSource, org.apache.hadoop.security.token.DelegationTokenIssuer
public class GoogleHadoopFileSystem extends org.apache.hadoop.fs.FileSystem implements org.apache.hadoop.fs.statistics.IOStatisticsSource
GoogleHadoopFileSystem is rooted in a single bucket at initialization time; in this case, Hadoop paths no longer correspond directly to general GCS paths, and all Hadoop operations going through this FileSystem will never touch any GCS bucket other than the bucket on which this FileSystem is rooted.

This implementation sacrifices a small amount of cross-bucket interoperability in favor of more straightforward FileSystem semantics and compatibility with existing Hadoop applications. In particular, it is not subject to bucket-naming constraints, and files are allowed to be placed in root.

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static enum 
    GoogleHadoopFileSystem.GcsFileChecksumType
    Available GCS checksum types for use with GoogleHadoopFileSystemConfiguration.GCS_FILE_CHECKSUM_TYPE.
    static enum 
    GoogleHadoopFileSystem.GlobAlgorithm
    Available GCS glob algorithms for use with GoogleHadoopFileSystemConfiguration.GCS_GLOB_ALGORITHM.

    Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

    org.apache.hadoop.fs.FileSystem.DirectoryEntries, org.apache.hadoop.fs.FileSystem.DirListingIterator<T extends org.apache.hadoop.fs.FileStatus>, org.apache.hadoop.fs.FileSystem.Statistics

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final String
    SCHEME
    URI scheme for GoogleHadoopFileSystem

    Fields inherited from class org.apache.hadoop.fs.FileSystem

    DEFAULT_FS, FS_DEFAULT_NAME_KEY, LOG, SHUTDOWN_HOOK_PRIORITY, statistics, TRASH_PREFIX, USER_HOME_PREFIX

    Fields inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer

    TOKEN_LOG

  • Constructor Summary

    Constructors
    Constructor
    Description
    GoogleHadoopFileSystem()
    Constructs an instance of GoogleHadoopFileSystem; the internal GoogleCloudStorageFileSystem will be set up with config settings when initialize() is called.

  • Method Summary

    Modifier and Type
    Method
    Description
    org.apache.hadoop.fs.FSDataOutputStream
    append(org.apache.hadoop.fs.Path hadoopPath, int bufferSize, org.apache.hadoop.util.Progressable progress)
    Appends to an existing file (optional operation).
    protected void
    checkPath(org.apache.hadoop.fs.Path path)
     
    void
    close()
     
    void
    completeLocalOutput(org.apache.hadoop.fs.Path fsOutputFile, org.apache.hadoop.fs.Path tmpLocalFile)
     
    void
    concat(org.apache.hadoop.fs.Path tgt, org.apache.hadoop.fs.Path[] srcs)
    Concat existing files into one file.
    void
    copyFromLocalFile(boolean delSrc, boolean overwrite, org.apache.hadoop.fs.Path[] srcs, org.apache.hadoop.fs.Path dst)
     
    void
    copyFromLocalFile(boolean delSrc, boolean overwrite, org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)
     
    void
    copyToLocalFile(boolean delSrc, org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)
     
    org.apache.hadoop.fs.FSDataOutputStream
    create(org.apache.hadoop.fs.Path hadoopPath, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)
     
    org.apache.hadoop.fs.FSDataOutputStream
    createNonRecursive(org.apache.hadoop.fs.Path hadoopPath, org.apache.hadoop.fs.permission.FsPermission permission, EnumSet<org.apache.hadoop.fs.CreateFlag> flags, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)
     
    boolean
    delete(org.apache.hadoop.fs.Path hadoopPath, boolean recursive)
     
    boolean
    deleteOnExit(org.apache.hadoop.fs.Path f)
     
    boolean
    exists(org.apache.hadoop.fs.Path f)
     
    String
    getCanonicalServiceName()
    org.apache.hadoop.fs.ContentSummary
    getContentSummary(org.apache.hadoop.fs.Path f)
     
    long
    getDefaultBlockSize()
     
    protected int
    getDefaultPort()
    The default port is listed as -1 as an indication that ports are not used.
    short
    getDefaultReplication()
    Gets the default replication factor.
    org.apache.hadoop.security.token.Token<?>
    getDelegationToken(String renewer)
     
    org.apache.hadoop.fs.FileChecksum
    getFileChecksum(org.apache.hadoop.fs.Path hadoopPath)
     
    org.apache.hadoop.fs.FileStatus
    getFileStatus(org.apache.hadoop.fs.Path hadoopPath)
     
    org.apache.hadoop.fs.FileStatus
    getFileStatusWithHint(org.apache.hadoop.fs.Path hadoopPath, org.apache.hadoop.conf.Configuration hint)
    Gets FileStatus with Hint.
    GoogleCloudStorageFileSystem
    getGcsFs()
    Gets GCS FS instance.
    GhfsGlobalStorageStatistics
    getGlobalGcsStorageStatistics()
    Get the storage statistics of this filesystem.
    org.apache.hadoop.fs.Path
    getHomeDirectory()
    Returns home directory of the current user.
    GhfsInstrumentation
    getInstrumentation()
     
    org.apache.hadoop.fs.statistics.IOStatistics
    getIOStatistics()
    Get the instrumentation's IOStatistics.
    String
    getScheme()
     
    GhfsStorageStatistics
    getStorageStatistics()
    Get the storage statistics of this filesystem.
    URI
    getUri()
    Returns a URI of the root of this FileSystem.
    long
    getUsed()
     
    Supplier<VectoredIOImpl>
    getVectoredIOSupplier()
     
    org.apache.hadoop.fs.Path
    getWorkingDirectory()
    Gets the current working directory.
    byte[]
    getXAttr(org.apache.hadoop.fs.Path path, String name)
     
    Map<String,byte[]>
    getXAttrs(org.apache.hadoop.fs.Path path)
     
    Map<String,byte[]>
    getXAttrs(org.apache.hadoop.fs.Path path, List<String> names)
     
    org.apache.hadoop.fs.FileStatus[]
    globStatus(org.apache.hadoop.fs.Path pathPattern)
    Returns an array of FileStatus objects whose path names match pathPattern.
    org.apache.hadoop.fs.FileStatus[]
    globStatus(org.apache.hadoop.fs.Path pathPattern, org.apache.hadoop.fs.PathFilter filter)
     
    boolean
    hasPathCapability(org.apache.hadoop.fs.Path path, String capability)
     
    void
    initialize(URI path, org.apache.hadoop.conf.Configuration config)
     
    org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>
    listLocatedStatus(org.apache.hadoop.fs.Path f)
     
    org.apache.hadoop.fs.FileStatus[]
    listStatus(org.apache.hadoop.fs.Path hadoopPath)
     
    org.apache.hadoop.fs.FileStatus[]
    listStatusStartingFrom(org.apache.hadoop.fs.Path startFrom)
    Gets FileStatus of all files which are lexicographically greater than and equal the provided path.
    List<String>
    listXAttrs(org.apache.hadoop.fs.Path path)
     
    org.apache.hadoop.fs.Path
    makeQualified(org.apache.hadoop.fs.Path path)
    Overridden to make root its own parent.
    boolean
    mkdirs(org.apache.hadoop.fs.Path hadoopPath, org.apache.hadoop.fs.permission.FsPermission permission)
     
    org.apache.hadoop.fs.FSDataInputStream
    open(org.apache.hadoop.fs.FileStatus status)
    This is an experimental API and can change without notice.
    org.apache.hadoop.fs.FSDataInputStream
    open(org.apache.hadoop.fs.Path hadoopPath, int bufferSize)
     
    CompletableFuture<org.apache.hadoop.fs.FSDataInputStream>
    openFileWithOptions(org.apache.hadoop.fs.Path hadoopPath, org.apache.hadoop.fs.impl.OpenFileParameters parameters)
    Initiate the open operation.
    protected void
    processDeleteOnExit()
     
    void
    removeXAttr(org.apache.hadoop.fs.Path path, String name)
    boolean
    rename(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)
     
    void
    setOwner(org.apache.hadoop.fs.Path p, String username, String groupname)
     
    void
    setPermission(org.apache.hadoop.fs.Path p, org.apache.hadoop.fs.permission.FsPermission permission)
     
    void
    setTimes(org.apache.hadoop.fs.Path p, long mtime, long atime)
     
    void
    setVerifyChecksum(boolean verifyChecksum)
     
    void
    setWorkingDirectory(org.apache.hadoop.fs.Path hadoopPath)
     
    void
    setXAttr(org.apache.hadoop.fs.Path path, String name, byte[] value, EnumSet<org.apache.hadoop.fs.XAttrSetFlag> flags)
    org.apache.hadoop.fs.Path
    startLocalOutput(org.apache.hadoop.fs.Path fsOutputFile, org.apache.hadoop.fs.Path tmpLocalFile)
     

    Methods inherited from class org.apache.hadoop.fs.FileSystem

    access, append, append, append, append, appendFile, areSymlinksEnabled, cancelDeleteOnExit, canonicalizeUri, clearStatistics, closeAll, closeAllForUGI, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createBulkDelete, createDataInputStreamBuilder, createDataInputStreamBuilder, createDataOutputStreamBuilder, createFile, createMultipartUploader, createNewFile, createNonRecursive, createNonRecursive, createPathHandle, createSnapshot, createSnapshot, createSymlink, delete, deleteSnapshot, enableSymlinks, fixRelativePart, get, get, get, getAclStatus, getAdditionalTokenIssuers, getAllStatistics, getAllStoragePolicies, getBlockSize, getCanonicalUri, getChildFileSystems, getDefaultBlockSize, getDefaultReplication, getDefaultUri, getEnclosingRoot, getFileBlockLocations, getFileBlockLocations, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getPathHandle, getQuotaUsage, getReplication, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getStoragePolicy, getTrashRoot, getTrashRoots, getUsed, isDirectory, isFile, listCorruptFileBlocks, listFiles, listLocatedStatus, listStatus, listStatus, listStatus, listStatusBatch, listStatusIterator, mkdirs, mkdirs, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, msync, newInstance, newInstance, newInstance, newInstanceLocal, open, open, open, openFile, openFile, openFileWithOptions, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, removeAcl, removeAclEntries, removeDefaultAcl, rename, renameSnapshot, resolveLink, resolvePath, satisfyStoragePolicy, setAcl, setDefaultUri, setDefaultUri, setQuota, setQuotaByStorageType, setReplication, setStoragePolicy, setWriteChecksum, setXAttr, supportsSymlinks, truncate, unsetStoragePolicy

    Methods inherited from class org.apache.hadoop.conf.Configured

    getConf, setConf

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer

    addDelegationTokens

  • Field Details

  • Constructor Details

    • GoogleHadoopFileSystem

      public GoogleHadoopFileSystem()
      Constructs an instance of GoogleHadoopFileSystem; the internal GoogleCloudStorageFileSystem will be set up with config settings when initialize() is called.

  • Method Details

    • initialize

      public void initialize(URI path, org.apache.hadoop.conf.Configuration config) throws IOException
      Overrides:
      initialize in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • checkPath

      protected void checkPath(org.apache.hadoop.fs.Path path)
      Overrides:
      checkPath in class org.apache.hadoop.fs.FileSystem

    • getScheme

      public String getScheme()
      Overrides:
      getScheme in class org.apache.hadoop.fs.FileSystem

    • open

      public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path hadoopPath, int bufferSize) throws IOException
      Specified by:
      open in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • open

      public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.FileStatus status) throws IOException
      This is an experimental API and can change without notice.
      Throws:
      IOException

    • create

      public org.apache.hadoop.fs.FSDataOutputStream create(org.apache.hadoop.fs.Path hadoopPath, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress) throws IOException
      Specified by:
      create in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • createNonRecursive

      public org.apache.hadoop.fs.FSDataOutputStream createNonRecursive(org.apache.hadoop.fs.Path hadoopPath, org.apache.hadoop.fs.permission.FsPermission permission, EnumSet<org.apache.hadoop.fs.CreateFlag> flags, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress) throws IOException
      Overrides:
      createNonRecursive in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • rename

      public boolean rename(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst) throws IOException
      Specified by:
      rename in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • delete

      public boolean delete(org.apache.hadoop.fs.Path hadoopPath, boolean recursive) throws IOException
      Specified by:
      delete in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • listStatus

      public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path hadoopPath) throws IOException
      Specified by:
      listStatus in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • listStatusStartingFrom

      public org.apache.hadoop.fs.FileStatus[] listStatusStartingFrom(org.apache.hadoop.fs.Path startFrom) throws IOException
      Gets FileStatus of all files which are lexicographically greater than and equal the provided path. It filters out any directory objects present in underneath storage.

      This is an experimental API can change without notice.

      Throws:
      IOException

    • mkdirs

      public boolean mkdirs(org.apache.hadoop.fs.Path hadoopPath, org.apache.hadoop.fs.permission.FsPermission permission) throws IOException
      Specified by:
      mkdirs in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • getFileStatus

      public org.apache.hadoop.fs.FileStatus getFileStatus(org.apache.hadoop.fs.Path hadoopPath) throws IOException
      Specified by:
      getFileStatus in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • getFileStatusWithHint

      public org.apache.hadoop.fs.FileStatus getFileStatusWithHint(org.apache.hadoop.fs.Path hadoopPath, org.apache.hadoop.conf.Configuration hint) throws IOException
      Gets FileStatus with Hint. Can be used if the caller want to pass the path type (file vs directory) hint. This hint can be used to prioritize GCS API calls inorder to improve performance and reduce redundant API calls without compromising performance and API behaviour. Currently, only "file" type hint is supported.

      This is an experimental API can change without notice.

      Throws:
      IOException

    • globStatus

      public org.apache.hadoop.fs.FileStatus[] globStatus(org.apache.hadoop.fs.Path pathPattern, org.apache.hadoop.fs.PathFilter filter) throws IOException
      Overrides:
      globStatus in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • getDelegationToken

      public org.apache.hadoop.security.token.Token<?> getDelegationToken(String renewer) throws IOException
      Specified by:
      getDelegationToken in interface org.apache.hadoop.security.token.DelegationTokenIssuer
      Overrides:
      getDelegationToken in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • copyFromLocalFile

      public void copyFromLocalFile(boolean delSrc, boolean overwrite, org.apache.hadoop.fs.Path[] srcs, org.apache.hadoop.fs.Path dst) throws IOException
      Overrides:
      copyFromLocalFile in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • copyFromLocalFile

      public void copyFromLocalFile(boolean delSrc, boolean overwrite, org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst) throws IOException
      Overrides:
      copyFromLocalFile in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • getFileChecksum

      public org.apache.hadoop.fs.FileChecksum getFileChecksum(org.apache.hadoop.fs.Path hadoopPath) throws IOException
      Overrides:
      getFileChecksum in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • exists

      public boolean exists(org.apache.hadoop.fs.Path f) throws IOException
      Overrides:
      exists in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • listLocatedStatus

      public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listLocatedStatus(org.apache.hadoop.fs.Path f) throws IOException
      Overrides:
      listLocatedStatus in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • getXAttr

      public byte[] getXAttr(org.apache.hadoop.fs.Path path, String name) throws IOException
      Overrides:
      getXAttr in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • getXAttrs

      public Map<String,byte[]> getXAttrs(org.apache.hadoop.fs.Path path) throws IOException
      Overrides:
      getXAttrs in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • getXAttrs

      public Map<String,byte[]> getXAttrs(org.apache.hadoop.fs.Path path, List<String> names) throws IOException
      Overrides:
      getXAttrs in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • listXAttrs

      public List<String> listXAttrs(org.apache.hadoop.fs.Path path) throws IOException
      Overrides:
      listXAttrs in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • getGlobalGcsStorageStatistics

      public GhfsGlobalStorageStatistics getGlobalGcsStorageStatistics()
      Get the storage statistics of this filesystem.
      Returns:
      the storage statistics

    • getStorageStatistics

      public GhfsStorageStatistics getStorageStatistics()
      Get the storage statistics of this filesystem.
      Overrides:
      getStorageStatistics in class org.apache.hadoop.fs.FileSystem
      Returns:
      the storage statistics

    • getIOStatistics

      public org.apache.hadoop.fs.statistics.IOStatistics getIOStatistics()
      Get the instrumentation's IOStatistics.
      Specified by:
      getIOStatistics in interface org.apache.hadoop.fs.statistics.IOStatisticsSource

    • getInstrumentation

      public GhfsInstrumentation getInstrumentation()

    • makeQualified

      public org.apache.hadoop.fs.Path makeQualified(org.apache.hadoop.fs.Path path)
      Overridden to make root its own parent. This is POSIX compliant, but more importantly guards against poor directory accounting in the PathData class of Hadoop 2's FsShell.
      Overrides:
      makeQualified in class org.apache.hadoop.fs.FileSystem

    • getUri

      public URI getUri()
      Returns a URI of the root of this FileSystem.
      Specified by:
      getUri in class org.apache.hadoop.fs.FileSystem

    • getDefaultPort

      protected int getDefaultPort()
      The default port is listed as -1 as an indication that ports are not used.
      Overrides:
      getDefaultPort in class org.apache.hadoop.fs.FileSystem

    • hasPathCapability

      public boolean hasPathCapability(org.apache.hadoop.fs.Path path, String capability)
      Specified by:
      hasPathCapability in interface org.apache.hadoop.fs.PathCapabilities
      Overrides:
      hasPathCapability in class org.apache.hadoop.fs.FileSystem

    • openFileWithOptions

      public CompletableFuture<org.apache.hadoop.fs.FSDataInputStream> openFileWithOptions(org.apache.hadoop.fs.Path hadoopPath, org.apache.hadoop.fs.impl.OpenFileParameters parameters) throws IOException
      Initiate the open operation. This is invoked from both the FileSystem and FileContext APIs
      Overrides:
      openFileWithOptions in class org.apache.hadoop.fs.FileSystem
      Parameters:
      hadoopPath - path to the file
      parameters - open file parameters from the builder.
      Returns:
      a future which will evaluate to the opened file.
      Throws:
      IOException - failure to resolve the link.
      IllegalArgumentException - unknown mandatory key

    • append

      public org.apache.hadoop.fs.FSDataOutputStream append(org.apache.hadoop.fs.Path hadoopPath, int bufferSize, org.apache.hadoop.util.Progressable progress) throws IOException
      Appends to an existing file (optional operation). Not supported.
      Specified by:
      append in class org.apache.hadoop.fs.FileSystem
      Parameters:
      hadoopPath - The existing file to be appended.
      bufferSize - The size of the buffer to be used.
      progress - For reporting progress if it is not null.
      Returns:
      A writable stream.
      Throws:
      IOException - if an error occurs.

    • concat

      public void concat(org.apache.hadoop.fs.Path tgt, org.apache.hadoop.fs.Path[] srcs) throws IOException
      Concat existing files into one file.
      Overrides:
      concat in class org.apache.hadoop.fs.FileSystem
      Parameters:
      tgt - the path to the target destination.
      srcs - the paths to the sources to use for the concatenation.
      Throws:
      IOException - IO failure

    • getWorkingDirectory

      public org.apache.hadoop.fs.Path getWorkingDirectory()
      Gets the current working directory.
      Specified by:
      getWorkingDirectory in class org.apache.hadoop.fs.FileSystem
      Returns:
      The current working directory.

    • getDefaultReplication

      public short getDefaultReplication()
      Gets the default replication factor.
      Overrides:
      getDefaultReplication in class org.apache.hadoop.fs.FileSystem

    • globStatus

      public org.apache.hadoop.fs.FileStatus[] globStatus(org.apache.hadoop.fs.Path pathPattern) throws IOException
      Returns an array of FileStatus objects whose path names match pathPattern.

      Return null if pathPattern has no glob and the path does not exist. Return an empty array if pathPattern has a glob and no path matches it.

      Overrides:
      globStatus in class org.apache.hadoop.fs.FileSystem
      Parameters:
      pathPattern - A regular expression specifying the path pattern.
      Returns:
      An array of FileStatus objects.
      Throws:
      IOException - if an error occurs.

    • getHomeDirectory

      public org.apache.hadoop.fs.Path getHomeDirectory()
      Returns home directory of the current user.

      Note: This directory is only used for Hadoop purposes. It is not the same as a user's OS home directory.

      Overrides:
      getHomeDirectory in class org.apache.hadoop.fs.FileSystem

    • getCanonicalServiceName

      public String getCanonicalServiceName()

      Returns the service if delegation tokens are configured, otherwise, null.

      Specified by:
      getCanonicalServiceName in interface org.apache.hadoop.security.token.DelegationTokenIssuer
      Overrides:
      getCanonicalServiceName in class org.apache.hadoop.fs.FileSystem

    • getGcsFs

      public GoogleCloudStorageFileSystem getGcsFs()
      Gets GCS FS instance.

    • getVectoredIOSupplier

      public Supplier<VectoredIOImpl> getVectoredIOSupplier()

    • deleteOnExit

      public boolean deleteOnExit(org.apache.hadoop.fs.Path f) throws IOException
      Overrides:
      deleteOnExit in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • processDeleteOnExit

      protected void processDeleteOnExit()
      Overrides:
      processDeleteOnExit in class org.apache.hadoop.fs.FileSystem

    • getContentSummary

      public org.apache.hadoop.fs.ContentSummary getContentSummary(org.apache.hadoop.fs.Path f) throws IOException
      Overrides:
      getContentSummary in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • copyToLocalFile

      public void copyToLocalFile(boolean delSrc, org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst) throws IOException
      Overrides:
      copyToLocalFile in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • startLocalOutput

      public org.apache.hadoop.fs.Path startLocalOutput(org.apache.hadoop.fs.Path fsOutputFile, org.apache.hadoop.fs.Path tmpLocalFile) throws IOException
      Overrides:
      startLocalOutput in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • completeLocalOutput

      public void completeLocalOutput(org.apache.hadoop.fs.Path fsOutputFile, org.apache.hadoop.fs.Path tmpLocalFile) throws IOException
      Overrides:
      completeLocalOutput in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • close

      public void close() throws IOException
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Overrides:
      close in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • getUsed

      public long getUsed() throws IOException
      Overrides:
      getUsed in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • getDefaultBlockSize

      public long getDefaultBlockSize()
      Overrides:
      getDefaultBlockSize in class org.apache.hadoop.fs.FileSystem

    • setWorkingDirectory

      public void setWorkingDirectory(org.apache.hadoop.fs.Path hadoopPath)
      Specified by:
      setWorkingDirectory in class org.apache.hadoop.fs.FileSystem

    • setVerifyChecksum

      public void setVerifyChecksum(boolean verifyChecksum)
      Overrides:
      setVerifyChecksum in class org.apache.hadoop.fs.FileSystem

    • setPermission

      public void setPermission(org.apache.hadoop.fs.Path p, org.apache.hadoop.fs.permission.FsPermission permission) throws IOException
      Overrides:
      setPermission in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • setOwner

      public void setOwner(org.apache.hadoop.fs.Path p, String username, String groupname) throws IOException
      Overrides:
      setOwner in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • setTimes

      public void setTimes(org.apache.hadoop.fs.Path p, long mtime, long atime) throws IOException
      Overrides:
      setTimes in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • setXAttr

      public void setXAttr(org.apache.hadoop.fs.Path path, String name, byte[] value, EnumSet<org.apache.hadoop.fs.XAttrSetFlag> flags) throws IOException
      Overrides:
      setXAttr in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • removeXAttr

      public void removeXAttr(org.apache.hadoop.fs.Path path, String name) throws IOException
      Overrides:
      removeXAttr in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException